Maintain Elasticsearch


Check the /var/log/elasticsearch/elasticsearch.log and /var/log/elasticsearch/elasticsearch_server.json log files for non-INFO messages:

grep -v INFO /var/log/elasticsearch/elasticsearch.log /var/log/elasticsearch/elasticsearch_server.json

Check the log files in the /var/log/elasticsearch directory, including:

  • elasticsearch_deprecation.log
  • elasticsearch_deprecation.json

Errors are logged in /var/log/elasticsearch/elasticsearch.log, for example:

  • “All shards failed for phase: [query]”, often followed by:
    • Failed to parse query [query]
    • Cannot parse ‘query
    • Failed to execute [SearchRequest{ … “query”:”query” … }]


/etc/elasticsearch/ configures a log file rotation strategy of:

appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB

To change to a 7-day rotation strategy, update this repository to replace this with:

appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified
appender.rolling.strategy.action.condition.nested_condition.age = 7D

Manage data

One-time setup

Set the password of the manage user in a netrc file, replacing PASSWORD:

echo 'machine login manage password PASSWORD' >> ~/.netrc

List indices:

curl -n

List base URLs in a given index, for example:

curl -n -X GET '' \
-H 'Content-Type: application/json' \
-d '{"aggs": {"base_urls": {"terms": {"field": "base_url", "size": 10000}}}}'

Delete documents matching a base URL:

curl -n -X POST '' \
-H 'Content-Type: application/json' \
-d '{"query": {"term": {"base_url": ""}}}'

Expire documents using OCDS Index:

ocdsindex expire --exclude-file=ocdsindex-exclude.txt

Search documents in a given index matching a base URL, for example:

curl -n -X GET '' \
-H 'Content-Type: application/json' \
-d '{"query": {"term": {"base_url": ""}}}'

List users’ queries:

zgrep -Eoh "q=[^&]+&" /var/log/apache2/* | grep -v '=test&' | grep -v '=tender&' | sort


If the ReadOnlyREST plugin is used:

  1. Get the ReadOnlyREST plugin’s ZIP file:

    1. Open the download page
    2. Select “Free Elasticsearch plugin” from the Select Product dropdown
    3. Select the Elasticsearch version from the Elastic Stack Version dropdown
    4. Enter your email address in Send to email
    5. Check Notify me about new versions and security fixes
    6. Click the Get it now button


    A new version might not yet be available for download. You can check the changelog.

  2. Move the ZIP file to the salt/private/files directory.

  3. Stop Elasticsearch, for example:

    ./ 'docs' service.stop elasticsearch
  4. Uninstall ReadOnlyREST, for example:

    ./ 'docs' "/usr/share/elasticsearch/bin/elasticsearch-plugin remove readonlyrest"
  5. Update readonlyrest_version and source_hash in the salt/elasticsearch/plugins/readonlyrest.sls file

  6. Deploy the service

Reference: Upgrading the plugin