Try out new MMapDirectory implementation in Lucene using Panama (FFM)

[austintlee]

Is your feature request related to a problem? Please describe.

I came across this talk by @uschindler. The latest on the work that Uwe presented is here. I am personally interested in seeing how this would work with the effort in Netty 5 to replace the use of the off-heap direct buffer with Panama. I think Uwe's work can be exploited in OpenSearch in all index read situations. But I think it might have a significant impact in shard or segment re-allocation or replication scenarios and for that, I am wondering if it would be possible to bypass the JVM heap entirely and pass references to mmap directly to the network module (Netty) via a foreign method. And if that is in fact doable, as was mentioned in the talk, how would unmap work when a network thread is still reading from a mmapped region.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[andrross]

[Triage - attendees 1 2 3]
@austintlee Thanks for filing this issue. Looking forward to seeing more prototyping here. I will also note that this will require adopting Netty 5 as described in #2619.

[uschindler]

I don't fully understand the issue. If OpenSearch uses MMapDirectory it automatically uses the new approach. So basically if you run OpenSearch with Lucene 9.5+ (I think its that version) and Java 19+ it will automatically use Panama. Starting with Lucene 9.10 it works with ALL versions Java 19, 20, 21, 22 and later! (final PR: apache/lucene#12706)

If there are other off-heap buffers in OpenSearch those could use a separate MR-JAR in Opensearch or reuse Netty stuff.

Lucene works without enabling the --enable-preview at runtime, because we use some compilation tricks.

P.S.: Lucene 10 (Java 21+) no longer has ByteBufferIndexInput and MMapDirectory is fully backed by Panama (see apache/lucene#13146)