LUCENE-9850: Use PFOR encoding for doc IDs (instead of FOR)

[gsmiller]

Description

Switch over to PFOR encoding for doc IDs (instead of FOR) to achieve better index compression.

Solution

Details are in the Jira issue, but I explored the index size vs. decompression speed tradeoffs using luceneutil benchmarks and found ~3.3% index size reduction with no significant OPS impact.

Tests

In addition to benchmarks, I ported over the PForDeltaUtil unit tests to ensure unit test coverage.

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Lucene maintainers access to contribute to my PR branch. (optional but recommended)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.

[jpountz]

I left a few comments but this looks great in general.

[jpountz]

I think we leave the implementation as is and hope that we can do something better with more explicit vectorization support in the future

(Sorry replying here as Github prevents me from replying on the existing thread)

+1 Let go with whichever of arr[i] = IDENTITY_PLUS_ONE[i] * val + base or arr[i] = (i+1) * val + base runs fastest in your micro benchmark. We can still improve things later if we find a way to trick the JVM into auto-vectorizing this loop.

[gsmiller]

@jpountz

+1 Let go with whichever of arr[i] = IDENTITY_PLUS_ONE[i] * val + base or arr[i] = (i+1) * val + base runs fastest in your micro benchmark. We can still improve things later if we find a way to trick the JVM into auto-vectorizing this loop.

Perfect, thanks! I'm changing this back to (i + 1) * val + base because it (somewhat surprisingly maybe, but I suppose this simple addition could be more efficient than an array reference) does consistently perform slightly better in microbenchmarks (arraryRef == 0 is this implementation while arrayRef == 1 references IDENTITY_PLUS_ONE[i]):

Benchmark                                        (arrayRef)  (bitsPerValue)  (exceptionCount)  (sameVal)   Mode  Cnt  Score   Error   Units
PackedIntsDeltaDecodeBenchmark.pForDeltaDecoder           0               0                 0          2  thrpt   20  7.915 ± 0.008  ops/us
PackedIntsDeltaDecodeBenchmark.pForDeltaDecoder           1               0                 0          2  thrpt   20  7.695 ± 0.010  ops/us

[gsmiller]

@jpountz I think I've addressed all of your feedback at this point. No rush if you've got other work occupying your time right now of course, just wanted to check in and make sure you're not waiting on me to make some additional changes. Thanks again for all your feedback!

[jpountz]

This looks great. I'll merge soon.

[rmuir]

Thanks @gsmiller ! I'm not really competent to review this stuff, just here for moral support. But the numbers look good to me.

[gsmiller]

Thanks @jpountz, @rmuir !