Data support tasks¶

Create a data support main server¶

Notify RBC Group of the new domain name for the new PostgreSQL server
Adjust reserved disk space to 1% for large disks:
```
tune2fs -m 1 /dev/md2
```

Dependent services¶

Redash

Add or update the Redash data source for the new PostgreSQL server
Add the Redash data source, if new, to the unrestricted Redash group

Tinyproxy

Update the allowed IP addresses in the pillar/tinyproxy.sls file
Deploy the docs service, when ready

Replica, if applicable

Update the allowed IP addresses and hostname in the pillar/kingfisher_replica.sls file
Deploy the kingfisher-replica service, when ready

Docker¶

Check that docker.uid in the pillar/kingfisher_main.sls file matches the entry in the /etc/passwd file.

Docker apps¶

Run migrations for Docker apps as the deployer user:

su - deployer

cd /data/deploy/kingfisher-process/
docker compose run --rm web python manage.py migrate

cd /data/deploy/pelican-frontend/
docker compose run --rm web python manage.py migrate

Pull new images and start new containers for each Docker app.

Pelican backend¶

Attention

A cron job updates the exchange_rates table every 12 hours. Complete this step before the next run.

The initial migrations for Pelican backend are run by Salt.

Connect to the old server, and dump the exchange_rates table:

sudo -i -u postgres psql -c '\copy exchange_rates (valid_on, rates, created, modified) to stdout' pelican_backend > exchange_rates.csv

Copy the database dump to your local machine. For example:

rsync -avz root@ocp13.open-contracting.org:~/exchange_rates.csv .

Copy the database dump to the new server. For example:

rsync -avz exchange_rates.sql root@ocp23.open-contracting.org:~/

Populate the exchange_rates table:

psql -U pelican_backend -h localhost -c "\copy exchange_rates (valid_on, rates, created, modified) from 'exchange_rates.csv';" pelican_backend

Kingfisher Collect¶

SSH into the new server as the incremental user:
1. Generate an SSH key pair:
```
ssh-keygen -t rsa -b 4096 -C "incremental"
```
2. Get the public SSH key:
```
cat ~/.ssh/id_rsa.pub
```
Add the public SSH key to the ssh.incremental list in the pillar/kingfisher_main.sls file:
```
ssh:
  incremental:
    - ssh-rsa AAAB3N...
```
Change cron.present to cron.absent in the salt/kingfisher/collect/incremental.sls file.
Deploy the old server and the new server.
SSH into the old server as the incremental user:
1. Stop any processes started by the cron jobs.
2. Dump the kingfisher_collect database:
```
pg_dump -U kingfisher_collect -h localhost -f kingfisher_collect.sql kingfisher_collect
```

SSH into the new server as the incremental user.

Copy the database dump from the old server. For example:

rsync -avz incremental@ocp04.open-contracting.org:~/kingfisher_collect.sql .

Load the database dump:

psql -U kingfisher_collect -h localhost -f kingfisher_collect.sql kingfisher_collect

Copy the data directory from the old server. For example:

rsync -avz incremental@ocp04.open-contracting.org:/home/incremental/data/ /home/incremental/data/

Copy the logs directory from the old server. For example:

rsync -avz incremental@ocp04.open-contracting.org:/home/incremental/logs/ /home/incremental/logs/

Remove the public SSH key from the ssh.incremental list in the pillar/kingfisher_main.sls file.
Change cron.absent to cron.present in the salt/kingfisher/collect/incremental.sls file.
Deploy the new server.
Update spiders in Kingfisher Collect.

Create a data support replica server¶

Update postgres.replica_ipv4 (and postgres.replica_ipv6, if applicable) in the pillar/kingfisher_main.sls file.