feat: expose number of rows covered per delta

[chebbyChefNEQ]

This PR adds a field to index_stats where we now return the number of rows covered by each delta in num_indexed_rows_per_delta

TODO:

• tests

[wjones127]

I'm a little concerned about whether the result is useful if we don't even guarantee the order of the index deltas. Otherwise seems fine. Though I feel like it's not that hard for users to compute this themselves, as the fragment_bitmap and num_rows should both be exposed publicly, in Python and Rust.

[wjones127]

Is this implicitly in the same order that list_indices returns? Do we even guarantee an order there?

[chebbyChefNEQ]

I think we guarantee append order, let me check

[chebbyChefNEQ]

I'm a little concerned about whether the result is useful if we don't even guarantee the order of the index deltas. Otherwise seems fine. Though I feel like it's not that hard for users to compute this themselves, as the fragment_bitmap and num_rows should both be exposed publicly, in Python and Rust.

We don't expose the per delta fragment bitmap to python or rust. This is only a stop gap so we have something for calculating the "base" index rows. Eventually this should be something we can track natively in v3

[chebbyChefNEQ]

I'm a little concerned about whether the result is useful if we don't even guarantee the order of the index deltas. Otherwise seems fine. Though I feel like it's not that hard for users to compute this themselves, as the fragment_bitmap and num_rows should both be exposed publicly, in Python and Rust.

We don't expose the per delta fragment bitmap to python or rust. This is only a stop gap so we have something for calculating the "base" index rows. Eventually this should be something we can track natively in v3

Actually, looks like it's a pub trait let me see if this is exposed thru lancedb

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 87.17949% with 5 lines in your changes missing coverage. Please review.

Project coverage is 78.85%. Comparing base (add0f34) to head (b3069e0).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance/src/index.rs	87.17%	0 Missing and 5 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2979      +/-   ##
==========================================
+ Coverage   78.81%   78.85%   +0.03%     
==========================================
  Files         235      236       +1     
  Lines       73141    73581     +440     
  Branches    73141    73581     +440     
==========================================
+ Hits        57648    58019     +371     
- Misses      12513    12554      +41     
- Partials     2980     3008      +28     

Flag	Coverage Δ
unittests	78.85% <87.17%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[chebbyChefNEQ]

going to merge this for the time being as this saves us from drifting from OSS.