fix: limit/offset pushdown was not working correctly in the presence of deletions

[westonpace]

We need to push back the scan start and potentially grab more rows than requested to make sure we fulfill the request. For v2 files we could also convert the row range into multiple ranges instead of grabbing more rows than required. This might be something to investigate in the future.

[wjones127]

Does this work right if the range straddles two fragments? Do we care that it does?

[wjones127]

Oh nice, I wanted this myself.

[westonpace]

Does this work right if the range straddles two fragments? Do we care that it does?
@wjones127

The logic for that is in the scan node. We take the limit/offset for the dataset and chop it up into limit/offsets for each of the fragments. That logic was actually ok. We were accounting for deleted rows so if the the range was something like 1000...3000 and each fragment had 1000 rows and 100 deleted rows we would translate it into:

fragment 0: skip (skip 900 rows)
fragment 1: 100..900 (skip 100, take 800)
fragment 2: 0..900 (skip 0, take 900)
fragment 3: 0..300 (skip 0, take 300)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 97.75281% with 2 lines in your changes missing coverage. Please review.

Project coverage is 77.90%. Comparing base (c0e966c) to head (7a7d981).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance-core/src/utils/deletion.rs	66.66%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2895   +/-   ##
=======================================
  Coverage   77.90%   77.90%           
=======================================
  Files         231      231           
  Lines       70528    70613   +85     
  Branches    70528    70613   +85     
=======================================
+ Hits        54946    55014   +68     
- Misses      12465    12472    +7     
- Partials     3117     3127   +10     

Flag	Coverage Δ
unittests	77.90% <97.75%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.