Kingfisher

Kingfisher is a family of tools to collect, pre-process, summarize and query OCDS data.

This page is about internal use of these tools by the Open Contracting Partnership. For your personal use, see each tool’s documentation:

• Kingfisher Collect

• Kingfisher Process

• Kingfisher Summarize

For internal use, Kingfisher Collect, Process and Summarize are deployed to a main server.

The following pages describe specific tasks for each tool:

• Kingfisher Collect
  • Review a new publication
  • Access Scrapyd’s web interface
  • Collect data with Kingfisher Collect
  • Update spiders in Kingfisher Collect
  • Access Scrapyd’s crawl logs
  • Create a .netrc file
  • Data retention policy
• Kingfisher Process
  • Review log files
  • Load local data
  • Export compiled releases from the database as record packages
  • Data retention policy
• Kingfisher Summarize
  • Review log files
  • Data retention policy

Connect to servers

One-time setup

Ask a systems administrator to add your public SSH key to the ssh.kingfisher list in the pillar/kingfisher.sls file.

The ocdskfp user owns the deployments of Kingfisher Process and Kingfisher Summarize, and can read the data and log files of Kingfisher Collect.

Connect to the main server as the ocdskfp user, to use the command-line tools of Kingfisher Process and Kingfisher Summarize:

curl --silent --connect-timeout 1 process.kingfisher.open-contracting.org:8255 || true
ssh ocdskfp@process.kingfisher.open-contracting.org

This user has access to the jq, flatten-tool and ocdskit command-line tools.

Note

The ocdskfs user owns the deployment of Kingfisher Collect. Only automated scripts and system administrators should manually delete any data and log files.