feat!: support to customize tokenizer

[BubbleCal]

users can customize the tokenizer:

• language
• remove long words
• lower case
• stem
• remove stop words
• ascii folding
  solve feat: support accent insensitive full text search #2996

This introduces a breaking change: we used en_stem as default tokenizer before, which stems the words, but this PR switches the default tokenizer to be without stemming

[wjones127]

Could we also support the AsciiFoldingFilter as an option? This will allow normalizing text with accents.

[SaintBacchus]

nice work. will the pr have some chinese tokenizers?

[BubbleCal]

Could we also support the AsciiFoldingFilter as an option? This will allow normalizing text with accents.

Yeah, added

[BubbleCal]

nice work. will the pr have some chinese tokenizers?

No, the tokenizer is based on tantivy's, unfortunately it doesn't support Chinese/Japanese

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 60.48780% with 81 lines in your changes missing coverage. Please review.

Project coverage is 78.12%. Comparing base (f9024ce) to head (95d929a).
Report is 71 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/inverted/tokenizer.rs	42.55%	49 Missing and 5 partials ⚠️
rust/lance-index/src/scalar/inverted/index.rs	37.50%	12 Missing and 3 partials ⚠️
rust/lance-index/src/scalar.rs	11.11%	8 Missing ⚠️
rust/lance-index/src/scalar/inverted/builder.rs	94.87%	1 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2992      +/-   ##
==========================================
- Coverage   78.19%   78.12%   -0.08%     
==========================================
  Files         239      240       +1     
  Lines       76782    76943     +161     
  Branches    76782    76943     +161     
==========================================
+ Hits        60043    60112      +69     
- Misses      13669    13740      +71     
- Partials     3070     3091      +21     

Flag	Coverage Δ
unittests	78.12% <60.48%> (-0.08%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[wjones127]

I'm on the fence whether these should be true by default. I feel like for more english text, quality would be better if these are all true. But for use cases like searching over code, I think all of these should likely be false. False seems like an okay default then, but we should definitely highlight in the user guide that setting these all to true would make sense for prose.

[BubbleCal]

right, usually stemming can improve recall.
Setting them to false by default cause tantivy does so

[wjones127]

Do we store this anywhere? When I run optimize_indices(), will it use the same configuration as when I run create_scalar_index() with tokenizer configuration?

[BubbleCal]

yes we store it in the index, and optimize_indices would load the existing index, then it would use this configuration, see here

[wjones127]

BTW it would be cool to have a standalone Tokenizer object users could play with. Right now the tokenizer is a black box. But users might want to run search queries through it to debug. I could imagine in LanceDB we might have a nice UI to help users do this. It would be nice if you could do something like:

tokenizer = LanceTokenizer(
    lang="en",
    lowercase=True,
    ascii_fold=True
)
test_results = tokenizer.tokenize("Hello World!")
assert test_results == [
    { "text": "hello", "pos": 0 },
    { "text": "world", "pos": 1 },
]

dataset.create_scalar_index("INVERTED", tokenizer=tokenizer)

[BubbleCal]

BTW it would be cool to have a standalone Tokenizer object users could play with. Right now the tokenizer is a black box. But users might want to run search queries through it to debug. I could imagine in LanceDB we might have a nice UI to help users do this. It would be nice if you could do something like:

tokenizer = LanceTokenizer(
    lang="en",
    lowercase=True,
    ascii_fold=True
)
test_results = tokenizer.tokenize("Hello World!")
assert test_results == [
    { "text": "hello", "pos": 0 },
    { "text": "world", "pos": 1 },
]

dataset.create_scalar_index("INVERTED", tokenizer=tokenizer)

cool idea! it would be very useful for debugging, will add this in next PR

[wjones127]

Looks good.

Note, because there is a breaking change, we need to increment the minor version. You can either do that in this PR, or wait for Weston's PR (also a breaking change) to merge and rerun the CI job.

[westonpace]

(the mentioned PR is merged so should just need a rebase)