Skip to content

Commit

Permalink
Recommend lowering the default mmap readahead.
Browse files Browse the repository at this point in the history
This is a follow-up of a discussion on apache#13219. `mmap` has a higher readahead
than regular `read()` operations by default, e.g. 128kB instead of 16kB on my
Linux box. On indexes that exceed the size of the page cache, this may trigger
performance issues due to page cache trashing and additional page cache
contention. Rather than forcing `MMapDirectory` to use `MADV_RANDOM` on all
files, it would make more sense to configure a lower `mmap` readahead at the
system level, e.g. the same readahead value as `read()` operations use.
  • Loading branch information
jpountz committed Mar 27, 2024
1 parent 1f909ba commit 8f177d7
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,16 @@
* fragmented address space. If you get an {@link IOException} about mapping failed, it is
* recommended to reduce the chunk size, until it works.
*
* <p><b>NOTE</b>: On some platforms like Linux, mmap comes with a higher readahead than regular
* read() operations, e.g. 128kB for mmap reads and 16kB for regular reads. Such a high default
* readahead can hurt performance on indexes that exceed the size of the page cache, by triggering
* more page cache trashing as well as more contention on the page cache. For typical workloads
* involving Lucene indexes, it is recommended to update the default readahead of mmap to the same
* readahead that is used by regular read() operations. This can be done by calling
* <code>sudo blockdev --setra 32 &lt;filesystem&gt;</code>, where 32 is the number of 512 byte
* sectors to read ahead, ie. 16kB, and &lt;filesystem&gt; is the filesystem where Lucene indexes
* are stored.
*
* <p>This class supports preloading files into physical memory upon opening. This can help improve
* performance of searches on a cold page cache at the expense of slowing down opening an index. See
* {@link #setPreload(BiPredicate)} for more details.
Expand Down

0 comments on commit 8f177d7

Please sign in to comment.