Maintain Elasticsearch

Troubleshoot

Check the /var/log/elasticsearch/elasticsearch.log and /var/log/elasticsearch/elasticsearch_server.json log files for non-INFO messages:

grep -v INFO /var/log/elasticsearch/elasticsearch.log /var/log/elasticsearch/elasticsearch_server.json

Check the log files in the /var/log/elasticsearch directory, including:

• elasticsearch_deprecation.log

• elasticsearch_deprecation.json

Errors are logged in /var/log/elasticsearch/elasticsearch.log, for example:

• “All shards failed for phase: [query]”, often followed by:

  • Failed to parse query [query]

  • Cannot parse ‘query’

  • Failed to execute [SearchRequest{ … “query”:”query” … }]

Note

/etc/elasticsearch/log4j2.properties configures a log file rotation strategy of:

appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB

To change to a 7-day rotation strategy, update this repository to replace this with:

appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified
appender.rolling.strategy.action.condition.nested_condition.age = 7D

Manage data

One-time setup

Set the password of the manage user of the standard.open-contracting.org service in a ~/.netrc file.

List indices:

curl -n https://standard.open-contracting.org/search/_cat/indices

List base URLs in a given index, for example:

curl -n -X GET 'https://standard.open-contracting.org/search/ocdsindex_en/_search?size=0&pretty' \
--json '{"aggs": {"base_urls": {"terms": {"field": "base_url", "size": 10000}}}}'

Delete documents matching a base URL:

curl -n -X POST 'https://standard.open-contracting.org/search/ocdsindex_en/_delete_by_query' \
--json '{"query": {"term": {"base_url": "https://standard.open-contracting.org/staging/1.1-dev/"}}}'

Expire documents using OCDS Index (pip install ocdsindex):

ocdsindex expire https://standard.open-contracting.org:443/search/ --exclude-file=ocdsindex-exclude.txt

Search documents in a given index matching a base URL, for example:

curl -n -X GET 'https://standard.open-contracting.org/search/ocdsindex_en/_search?size=10000' \
--json '{"query": {"term": {"base_url": "https://standard.open-contracting.org/staging/1.1-dev/"}}}'

List users’ queries:

zgrep -Eoh "q=[^&]+&" /var/log/apache2/* | grep -v '=test&' | grep -v '=tender&' | sort

Upgrade

Note

Before upgrading Elasticsearch, check that all plugins (below) support the new version.

1. Add support for the new version to ocds-index.

2. If upgrading to a new major version:

   1. Update name under pkgrepo.managed in the salt/elasticsearch/init.sls file.

   2. Deploy the server.

3. Connect to the server as the root user.

4. Perform any outstanding updates:

   apt-get update && apt-get dist-upgrade
   

5. Update Elasticsearch (the Elasticsearch package is held to prevent accidental updates):

   apt-mark unhold elasticsearch
   apt-get update && apt-get dist-upgrade
   apt-mark hold elasticsearch
   

6. Update plugins, as described below.

7. Test Elasticsearch is working.

   1. Check that the service is running without errors.

      systemctl status elasticsearch
      

   2. Test the site search works.

ReadOnlyREST

If the ReadOnlyREST plugin is used:

1. Check its changelog for a new version. Note which versions of Elasticsearch are supported.

2. In the server’s Pillar file, set elasticsearch.plugins.readonlyrest.version to the version of ReadOnlyREST to install, and set elasticsearch.version to the already installed version of Elasticsearch:

   dpkg-query --show elasticsearch
   

3. Stop Elasticsearch, for example:

   systemctl stop elasticsearch
   

4. Uninstall ReadOnlyREST, for example:

   /usr/share/elasticsearch/bin/elasticsearch-plugin remove readonlyrest
   

5. Deploy the server

Reference: Upgrading the plugin