Configure Elasticsearch#
Check system configuration#
Elasticsearch has instructions under Important System Configuration, most of which don’t require any changes. To check if changes are needed:
-
ulimit -n
-
sysctl vm.max_map_count
-
ulimit -u
JNA temporary directory isn’t mounted with noexec:
grep tmp /etc/fstab
-
sysctl net.ipv4.tcp_retries2
Set swappiness value#
As recommended by Elasticsearch, add to your service’s Pillar file:
vm:
swappiness: 1
Enable public access#
As stated by Elasticsearch, “Do not expose Elasticsearch directly to users.” For the OCDS documentation, we use the ReadOnlyREST plugin to control access.
Add the
elasticsearch.plugins.readonlyrest
state file to your service’s target in thesalt/top.sls
file.Allow anyone to access Elasticsearch. Add to your service’s Pillar file:
elasticsearch: public_access: True
Allow cross-origin HTTP requests (optional). Add to your service’s Pillar file, for example:
elasticsearch: allowed_origins: https://standard.open-contracting.org
Configure ReadOnlyREST SSL certificates in your service’s Pillar file. If Apache and Elasticsearch serve content from the same domain, you can reuse the certificates acquired by the
mod_md
module. For example:elasticsearch: plugins: readonlyrest: certificate_key_file: /etc/apache2/md/domains/standard.open-contracting.org/privkey.pem certificate_file: /etc/apache2/md/domains/standard.open-contracting.org/pubcert.pem
If reusing certificates, configure the
mod_md
module to restart Elasticsearch after renewing certificates:apache: modules: mod_md: MDNotifyCmd: /opt/restart-elasticsearch.sh
Add users for public searches and for admin actions. Add to your service’s private Pillar file, replacing
AUTH_KEY_SHA512
with the output ofecho -n 'USERNAME:PASSWORD' | shasum -a 512
(replacingUSERNAME
andPASSWORD
with a strong password each time):elasticsearch: plugins: readonlyrest: users: - auth_key_sha512: AUTH_KEY_SHA512 username: public groups: - public - auth_key_sha512: AUTH_KEY_SHA512 username: manage groups: - manage
Test the public user, replacing
PASSWORD
. For example, for thestandard.open-contracting.org
domain:curl -u 'public:PASSWORD' https://standard.open-contracting.org:9200/ocdsindex_en/_search \ -H 'Content-Type: application/json' \ -d '{"query": {"term": {"base_url": "https://standard.open-contracting.org/staging/1.1-dev/"}}}'
Test the admin user, replacing
PASSWORD
. For example, for thestandard.open-contracting.org
domain:curl -u 'manage:PASSWORD' https://standard.open-contracting.org:9200/_cat/indices
Troubleshoot#
If a request gets a HTTP 4XX error, connect to the server, and run:
tail -f /var/log/elasticsearch/elasticsearch.log
You will see a message like (newlines are added for readability):
[2020-12-23T23:26:01,367][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [live.docs.opencontracting.uk0.bigv.io]
FORBIDDEN by default req={
ID:2016835989-238874394#2554,
TYP:GetIndexRequest,
CGR:N/A,
USR:manage (attempted),
BRS:true,
KDX:null,
ACT:indices:admin/get,
OA:174.89.151.140/32,
XFF:null,
DA:5.28.62.151/32,
IDX:ocdsindex_en,
MET:HEAD,
PTH:/ocdsindex_en,
CNT:<N/A>,
HDR:Accept=*/*, Authorization=<OMITTED>, Host=standard.open-contracting.org:9200, User-Agent=curl/7.64.1, content-length=0,
HIS:
[Allow localhost->
RULES:[hosts->false],
RESOLVED:[indices=ocdsindex_en]
],
[Allow the public group to search indices created by OCDS Index->
RULES:[groups->false],
RESOLVED:[indices=ocdsindex_en]
],
[Allow the manage group to manage indices created by OCDS Index->
RULES:[groups->true, actions->false],
RESOLVED:[user=manage;group=manage;av_groups=manage;indices=ocdsindex_en]
]
}
The relevant content is:
FORBIDDEN by default
means no access control (ACL) block matched the request.USR:
indicates the user to be matched by users or groups rules.ACT:
indicates the Elasticsearch action to be matched by actions rules.OA:
indicates the origin address to be matched by hosts rules.IDX:
indicates the Elasticsearch index to be matched by indices rules.MET:
indicates the HTTP method,PTH:
the URL path, andHDR:
the HTTP headers. Check thatAuthorization
is set.Note
While rules at the HTTP level are allowed, “please refrain from using HTTP level rules,” as documented by ReadOnlyREST.
HIS:
indicates which rules passed (true
) or failed (false
), and how values were resolved. This is the most relevant information for debugging ACL blocks.