Kingfisher Process

Read the Kingfisher Process documentation, which covers general usage.

Note

Is the service unresponsive or erroring? Follow these instructions.

Review log files

Kingfisher Process writes log messages to the /var/log/kingfisher.log file. The log file is rotated weekly; last week’s log file is at /var/log/kingfisher.log.1, and earlier log files are compressed at /var/log/kingfisher.log.2.gz, etc.

The log files can be read by the ocdskfp user, after connecting to the server.

Log messages are formatted as:

[date] [hostname] %(asctime)s - %(process)d - %(name)s - %(levelname)s - %(message)s

You can filter messages by topic. For example:

grep NAME /var/log/kingfisher.log | less

For more information, read Kingfisher Process’ logging documentation.

Load local data

  1. Connect to the main server as the ocdskfp user

  2. Change into the local-load directory:

    cd ~/local-load

  3. Create a data directory following the pattern source-YYYY-MM-DD-analyst. For example: moldova-2020-04-07-romina

    • If the data source is the same as for an existing spider, use the same source ID, for example: moldova. Otherwise, use a different source ID that follows our regular pattern country[_region][_label], for example: moldova_covid19.

  4. If you need to download an archive file (e.g. ZIP) from a remote URL, prefer curl to wget, because wget sometimes writes unwanted files like wget-log.

  5. Load the data, using the local-load command.

  6. Delete the data directory once you’re satisfied that it loaded correctly.

Data retention policy

On the first day of each month, the following are deleted:

  • Collections that ended over a year ago
  • Collections that never ended and started over 2 months ago
  • Collections that ended over 2 months ago and have no data