perf: parallelize ngram indexing

[BubbleCal]

total indexing time reduced from 23s to 5s

ngram_index(1000000)    time:   [5.1192 s 5.1756 s 5.2319 s]
                        change: [-78.163% -77.791% -77.410%] (p = 0.00 < 0.05)
                        Performance has improved.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 97.14286% with 1 line in your changes missing coverage. Please review.

Project coverage is 78.46%. Comparing base (33ae43b) to head (e520053).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/ngram.rs	97.14%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3501      +/-   ##
==========================================
- Coverage   78.48%   78.46%   -0.03%     
==========================================
  Files         252      252              
  Lines       94011    94078      +67     
  Branches    94011    94078      +67     
==========================================
+ Hits        73783    73815      +32     
- Misses      17232    17268      +36     
+ Partials     2996     2995       -1     

Flag	Coverage Δ
unittests	78.46% <97.14%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[westonpace]

I'm not entirely sure I understand the buffer_size calculation? How is it related to get_num_compute_intensive_cpus? Why not just used a small fixed number?

[BubbleCal]

tbh I forget why it's calculated as this, this is from FTS indexing code. But yes it looks like a small fixed numbers is fine here. set it to 2