Kingfisher Process

Read the Kingfisher Process documentation, which covers general usage.


Is the service unresponsive or erroring? Follow these instructions.

Review log files

Kingfisher Process writes log messages to the /var/log/kingfisher.log file. The log file is rotated weekly; last week’s log file is at /var/log/kingfisher.log.1, and earlier log files are compressed at /var/log/kingfisher.log.2.gz, etc.

The log files can be read by the ocdskfp user, after connecting to the server.

Log messages are formatted as:

[date] [hostname] %(asctime)s - %(process)d - %(name)s - %(levelname)s - %(message)s

You can filter messages by topic. For example:

grep NAME /var/log/kingfisher.log | less

For more information, read Kingfisher Process’ logging documentation.

Load local data

  1. Connect to the main server as the ocdskfp user

  2. Change into the local-load directory:

    cd ~/local-load
  3. Create a data directory following the pattern source-YYYY-MM-DD-analyst. For example: moldova-2020-04-07-romina

    • If the data source is the same as for an existing spider, use the same source ID, for example: moldova. Otherwise, use a different source ID that follows our regular pattern country[_region][_label], for example: moldova_covid19.
  4. If you need to download an archive file (e.g. ZIP) from a remote URL, prefer curl to wget, because wget sometimes writes unwanted files like wget-log.

  5. Load the data, using the local-load command.

  6. Delete the data directory once you’re satisfied that it loaded correctly.

Data retention policy

On the first day of each month, the following are deleted:

  • Collections that ended over a year ago
  • Collections that never ended and started over 2 months ago
  • Collections that ended over 2 months ago and have no data