Kingfisher

Kingfisher is a family of tools to collect, pre-process, summarize and query OCDS data.

This page is about internal use of these tools by the Open Contracting Partnership. For your personal use, see each tool’s documentation:

• Kingfisher Collect

• Kingfisher Process

• Kingfisher Summarize

For internal use, Kingfisher Collect, Process and Summarize are deployed to a data support server.

The following pages describe specific tasks for each tool:

• Kingfisher Collect
  • Review a new publication
  • Access Scrapyd’s web interface
  • Collect data with Kingfisher Collect
  • Update spiders in Kingfisher Collect
  • Access Scrapy’s crawl logs
  • Data retention policy
• Kingfisher Process
  • Load local data
  • Add structural checks
  • Remove a collection
  • Check on progress
    • Using the command-line interface
    • Using RabbitMQ
  • Export compiled releases from the database as record packages
  • Data retention policy
• Kingfisher Summarize
  • Summarize collections
  • Review log files
  • Data retention policy

Connect to servers

One-time setup

Request access from James or Yohanna. (They will need your public SSH key to add a key-value pair under the users key in the pillar/kingfisher_main.sls file.)

Connect to the data support server, replacing USER:

ssh USER@collect.kingfisher.open-contracting.org

In addition to the Kingfisher tools, users have access to these command-line tools:

• jq, to query JSON data

• ripgrep, to grep, but fast

• flatten-tool, to unflatten data for local load

• ocdskit, to transform data for local load

• unrar, to unarchive data for local load