Configure Elasticsearch#

Check system configuration#

Elasticsearch has instructions under Important System Configuration, most of which don’t require any changes. To check if changes are needed:

File descriptors:
```
ulimit -n
```
Virtual memory:
```
sysctl vm.max_map_count
```
Number of threads:
```
ulimit -u
```
JNA temporary directory isn’t mounted with noexec:
```
grep tmp /etc/fstab
```
TCP retransmission timeout:
```
sysctl net.ipv4.tcp_retries2
```

Set swappiness value#

As recommended by Elasticsearch, add to your service’s Pillar file:

vm:
  swappiness: 1

Enable public access#

As stated by Elasticsearch, “Do not expose Elasticsearch directly to users.” For the OCDS documentation, we use the ReadOnlyREST plugin to control access.

Add the elasticsearch.plugins.readonlyrest state file to your service’s target in the salt/top.sls file.
Allow anyone to access Elasticsearch. Add to your service’s Pillar file:
```
elasticsearch:
  public_access: True
```
Allow cross-origin HTTP requests (optional). Add to your service’s Pillar file, for example:
```
elasticsearch:
  allowed_origins: https://standard.open-contracting.org
```

Configure ReadOnlyREST SSL certificates in your service’s Pillar file. If Apache and Elasticsearch serve content from the same domain, you can reuse the certificates acquired by the mod_md module. For example:

elasticsearch:
  plugins:
    readonlyrest:
      certificate_key_file: /etc/apache2/md/domains/standard.open-contracting.org/privkey.pem
      certificate_file: /etc/apache2/md/domains/standard.open-contracting.org/pubcert.pem

If reusing certificates, configure the mod_md module to restart Elasticsearch after renewing certificates:

apache:
  modules:
    mod_md:
      MDNotifyCmd: /opt/restart-elasticsearch.sh

Add users for public searches and for admin actions. Add to your service’s private Pillar file, replacing AUTH_KEY_SHA512 with the output of echo -n 'USERNAME:PASSWORD' | shasum -a 512 (replacing USERNAME and PASSWORD with a strong password each time):

elasticsearch:
  plugins:
    readonlyrest:
      users:
        - auth_key_sha512: AUTH_KEY_SHA512
          username: public
          groups:
            - public
        - auth_key_sha512: AUTH_KEY_SHA512
          username: manage
          groups:
            - manage

Deploy the service

Test the public user, replacing PASSWORD. For example, for the standard.open-contracting.org domain:

curl -u 'public:PASSWORD' https://standard.open-contracting.org:9200/ocdsindex_en/_search \
-H 'Content-Type: application/json' \
-d '{"query": {"term": {"base_url": "https://standard.open-contracting.org/staging/1.1-dev/"}}}'

Test the admin user, replacing PASSWORD. For example, for the standard.open-contracting.org domain:
```
curl -u 'manage:PASSWORD' https://standard.open-contracting.org:9200/_cat/indices
```

Troubleshoot#

If a request gets a HTTP 4XX error, connect to the server, and run:

tail -f /var/log/elasticsearch/elasticsearch.log

You will see a message like (newlines are added for readability):

[2020-12-23T23:26:01,367][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [live.docs.opencontracting.uk0.bigv.io]
  FORBIDDEN by default req={
    ID:2016835989-238874394#2554,
    TYP:GetIndexRequest,
    CGR:N/A,
    USR:manage (attempted),
    BRS:true,
    KDX:null,
    ACT:indices:admin/get,
    OA:174.89.151.140/32,
    XFF:null,
    DA:5.28.62.151/32,
    IDX:ocdsindex_en,
    MET:HEAD,
    PTH:/ocdsindex_en,
    CNT:<N/A>,
    HDR:Accept=*/*, Authorization=<OMITTED>, Host=standard.open-contracting.org:9200, User-Agent=curl/7.64.1, content-length=0,
    HIS:
      [Allow localhost->
        RULES:[hosts->false],
        RESOLVED:[indices=ocdsindex_en]
      ],
      [Allow the public group to search indices created by OCDS Index->
        RULES:[groups->false],
        RESOLVED:[indices=ocdsindex_en]
      ],
      [Allow the manage group to manage indices created by OCDS Index->
        RULES:[groups->true, actions->false],
        RESOLVED:[user=manage;group=manage;av_groups=manage;indices=ocdsindex_en]
      ]
  }

The relevant content is:

FORBIDDEN by default means no access control (ACL) block matched the request.
USR: indicates the user to be matched by users or groups rules.
ACT: indicates the Elasticsearch action to be matched by actions rules.
OA: indicates the origin address to be matched by hosts rules.
IDX: indicates the Elasticsearch index to be matched by indices rules.
MET: indicates the HTTP method, PTH: the URL path, and HDR: the HTTP headers. Check that Authorization is set.

Note

While rules at the HTTP level are allowed, “please refrain from using HTTP level rules,” as documented by ReadOnlyREST.
HIS: indicates which rules passed (true) or failed (false), and how values were resolved. This is the most relevant information for debugging ACL blocks.