Document that Transparent Huge Pages should be disabled on Linux

[jakommo]

There seems to be a Kernel issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1644056 that causes a Kernel crash during high load.
Also reported in discuss: https://discuss.elastic.co/t/elasticsearch-5-4-2-process-periodically-dying-with-high-cpu-load-and-kernel-message-pgtable-generic-c-33-bad-pmd/92239

It seems like this can be worked around by disabling THP on linux, e.g. echo -n never > /sys/kernel/mm/transparent_hugepage/enabled.

I had a chat with @jasontedor and we should recommend to disable THP in general, not just with the effected Kernel versions.
Seems like this is enabled by default on at least Ubuntu 14/16.04 and RHEL 6/7.

I think Important System Configuration would be a good place for this.

[jakommo]

I talked to a user and they still experienced the above kernel bug after disabling THP, a lot less frequent though.
What seems to have solved it for them now is to also disable NUMA, so maybe we can add this as well (if there are no objections from dev side).

[jasontedor]

As I mentioned in another channel, we should make the THP recommendation independent of any kernel bugs that may or may not be present.

As far as NUMA, our recommendations here require gathering more data and running some experiments. I don’t think we should base our recommendations on the basis of one data point (that might be fixed in different kernel versions).

[elasticmachine]

Pinging @elastic/es-core-infra

[dliappis]

One additional data point here, me and @danielmitterdorfer are working on a) evaluating the stability and b) performance behavior of Elasticsearch with and without THP. To be more exact, tuning involves not only testing THP enabled/false but also the defrag THP option, which as of kernel 4.6.1 offers new defrag strategies.

So far in our nightly benchmarking environment we have discovered that disabling THP (which in newer kernels is usually done by setting /sys/kernel/mm/transparent_hugepage/{defrag,enabled} to madvise) causes a performance drop in Elasticsearch.

Additionally, more recent versions of the Ubuntu kernel (starting with 4.12.2) are now setting THP to madvise from the enabled which used to be the default, which is how we became aware of the performance regression in the first place. madvise will also be the default setting in the upcoming Ubuntu 18.04 LTS release in a few days.

We will be providing more details when the necessary longrunning benchmarks have finished, backed by enough CI runs plus sufficient benchmarking data for a THP suggestion.

[alexander-marquardt]

MongoDB recommends against THP in the following document, so the same logic might apply to Elasticsearch if our access patterns are similar: https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/

[pmoust]

@dliappis did you get the chance to test more / dig deeper on this?

[dliappis]

@pmoust unfortunately the deep dive work has been paused due to other activities. However, since my last comment some things haven't been documented so I'll summarize:

• The indexing performance drop in Elasticsearch when THP is set to madvise was observed only with the geonames track. The drop was approximately 7%. The performance didn't change in the remaining tracks.
• To evaluate stability impact with THP set to always, all workers in elasticsearch-ci.elastic.co have /sys/kernel/mm/transparent_hugepage/{defrag,enabled} set to always and we haven't observed any obvious impact in stability.
• All our nightly benchmark environments (see history of changes) have transparent_hugepage enabled and defrag settings set to always.

[jasontedor]

I wouldn't expect stability issues with always, just long-tail latency spikes from transparent huge page defragging.

[jrodewig]

[docs issue triage]

[elasticmachine]

Pinging @elastic/es-perf (Team:Performance)

[zez3]

@dliappis
Is there somewhere a detailed description or the Cluster nodes used in elasticsearch-ci.elastic.co testing?
I am most interested in JVM heap sizes and per node memory usage during the tests.
Did the nodes used swap?
How much swap was used?

[lockewritesdocs]

This issue hasn’t been updated in 3+ years so I’m closing it. We can revisit if needed.

[pmoust]

I'd like us to have a look again into this @tomcallahan @dliappis, if you agree.
It's unclear to me what the exact recommendation is nowadays.

[okwute419]

I'd like us to have a look again into this @tomcallahan @dliappis, if you agree. It's unclear to me what the exact recommendation is nowadays.

This issue hasn't been updated in almost 1yr, can we close this?

[fholzer]

Give the previous statement about CA. 7% performance degradation under certain circumstances I would assume it would be beneficial to get an officially recommendation on this topic an add the same.to the official documentation.

[elasticsearchmachine]

Pinging @elastic/es-docs (Team:Docs)

[geekpete]

Will the recent enhancements to Lucene 9.11.0 around madvise have any bearing on this issue once a stack release containing Lucene 9.11.0 is released? (8.14.0 uses Lucene 9.10.0)?

https://lucene.apache.org/core/9_11_0/changes/Changes.html#v9.11.0.new_features

• Add support for posix_madvise to Java 21 MMapDirectory apache/lucene#13196
• Use ReadAdvice#RANDOM when appropriate. apache/lucene#13222