Data support¶

Create a data support main server¶

Dependencies¶

Tinyproxy

Update the allowed IP addresses in the pillar/tinyproxy.sls file.
Deploy the docs service, when ready.

Replica, if applicable

Update the allowed IP addresses and hostname in the pillar/kingfisher_replica.sls file.
Deploy the kingfisher-replica service, when ready.

Dependents¶

Notify RBC Group of the new domain name for the new PostgreSQL server.

Update Salt and halt jobs¶

Check that docker.uid in the server’s Pillar file matches the entry in the /etc/passwd file for the docker.user (deployer).
Change cron.present to cron.absent in the salt/pelican/backend/init.sls file.
Comment out the postgres.backup section of the Pillar file.
Deploy the old server and the new server.
On the old server:
1. Delete the /etc/cron.d/postgres_backups file.
2. docker compose down all containers.
Check that no crawls are running at https://collect.kingfisher.open-contracting.org/jobs.

If a crawl is running, job owners can cancel jobs.
Check that no messages are enqueued at https://rabbitmq.kingfisher.open-contracting.org.

If a job is running in Kingfisher Process, job owners can cancel jobs.

Docker apps¶

Run migrations for Docker apps as the deployer user:

su - deployer

cd /data/deploy/kingfisher-process/
docker compose run --rm --name django-migrate cron python manage.py migrate

cd /data/deploy/pelican-frontend/
docker compose run --rm --name django-migrate web python manage.py migrate

Pull new images and start new containers for each Docker app.

Kingfisher Collect¶

Once DNS has propagated, Update spiders in Kingfisher Collect.

Copy incremental data¶

SSH into the new server as the incremental user:
1. Generate an SSH key pair:
```
ssh-keygen -t rsa -b 4096 -C "incremental"
```
2. Get the public SSH key:
```
cat ~/.ssh/id_rsa.pub
```
Add the public SSH key to the ssh.incremental list in the pillar/kingfisher_main.sls file:
```
ssh:
  incremental:
    - ssh-rsa AAAB3N...
```
Change cron.present to cron.absent in the salt/kingfisher/collect/incremental.sls file.
Deploy the old server and the new server.
SSH into the old server as the incremental user:
1. Stop any processes started by the cron jobs.
2. Dump the kingfisher_collect database:
```
pg_dump -U kingfisher_collect -h localhost -f kingfisher_collect.sql kingfisher_collect
```

SSH into the new server as the incremental user.

Copy the database dump from the old server. For example:

rsync -avz incremental@ocp04.open-contracting.org:~/kingfisher_collect.sql .

Load the database dump:

psql -U kingfisher_collect -h localhost -f kingfisher_collect.sql kingfisher_collect

Copy the data directory from the old server. For example:

rsync -avz incremental@ocp04.open-contracting.org:/home/incremental/data/ /home/incremental/data/

Copy the logs directory from the old server. For example:

rsync -avz incremental@ocp04.open-contracting.org:/home/incremental/logs/ /home/incremental/logs/

Remove the public SSH key from the ssh.incremental list in the pillar/kingfisher_main.sls file.
Change cron.absent to cron.present in the salt/kingfisher/collect/incremental.sls file.
Deploy the new server.

Pelican backend¶

The initial migrations for Pelican backend, which create the exchange_rates table, are run by Salt.

Connect to the old server, and dump the exchange_rates table:

sudo -i -u postgres psql -c '\copy exchange_rates (valid_on, rates, created, modified) to stdout' pelican_backend > exchange_rates.csv

Copy the database dump to your local machine. For example:

rsync -avz root@ocp13.open-contracting.org:~/exchange_rates.csv .

Copy the database dump to the new server. For example:

rsync -avz exchange_rates.sql root@ocp23.open-contracting.org:~/

Populate the exchange_rates table:

psql -U pelican_backend -h localhost -c "\copy exchange_rates (valid_on, rates, created, modified) from 'exchange_rates.csv';" pelican_backend

Restore Salt and start jobs¶

Change cron.absent to cron.present in the salt/pelican/backend/init.sls file.
Uncomment the postgres.backup section of the Pillar file.
Deploy the new server.

Create a data support replica server¶

Update postgres.replica_ipv4 (and postgres.replica_ipv6, if applicable) in the pillar/kingfisher_main.sls file.