fix: add versioning and bypass broken row counts

[wjones127]

Adds a new feature: WriterVersion in the manifest.

Also fixes two bugs:

• Incorrect physical_rows recorded when migrating from old versions of tables with deletions #1531 Adds bypass logic to read correct physical_rows when old version is detected.
  Will update the value on write as needed.
• When migrate_fragments runs, we do it on the new fragments and not the old,
  fixing a data loss issue.

Fixes #1531
Fixes #1535

[github-actions]

ACTION NEEDED

Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[wjones127]

I found another critical bug here 😞

We migrate the old fragments not the new, so this basically is rolling back the transactions changes. This means data loss in most cases.

[eddyxu]

Could we have some tests to cover this case?

[wjones127]

Working on adding tests now.

[wjones127]

Tests have been added.

[wjones127]

I'm confused right now. This seems like it would have caused new fragments to be ignored, but my repro in #1531 does not seem to cause this.

[wjones127]

Figured it out, the data loss bug was added in a later version.

[wjones127]

Validated this manually building off of repro in README:

Repro setup

Install pylance 0.7.5

python -m venv .venv
source .venv/bin/activate
pip install pylance==0.7.5
python

Write a table, and delete some rows

import lance
import pyarrow as pa

tab = pa.table({'x': range(100)})
dataset = lance.write_dataset(tab, 'test')
dataset.delete("x >= 10 and x < 20")
dataset.count_rows()

90

Now, install pylance 0.8.0:

pip install pylance==0.8.0
python

Count rows at first is correct. However, once we write to the table, it becomes incorrect:

import lance
import pyarrow as pa

dataset = lance.dataset('test')
dataset.count_rows() # 90

tab = pa.table({'x': range(10)})
dataset = lance.write_dataset(tab, 'test', mode='append')
dataset.count_rows() # 90 
dataset.to_table().num_rows # 100

Steps in this fix

Using feature branch:

import lance
dataset = lance.dataset("test")

dataset.count_rows()

100

import pyarrow as pa
tab = pa.table({'x': range(2)})
dataset = lance.write_dataset(tab, dataset.uri, mode="append")
dataset.count_rows()

102

Validating in old version

Back in pylance 0.8.0:

>>> import lance
>>> dataset = lance.dataset('test')
>>> dataset.count_rows()

102

[eddyxu]

in what condition that physical_rows = None?

[wjones127]

When the dataset was written prior to 0.8.0, this field is not filled in (because the field didn't exist then).

[wjones127]

BTW that change is more relevant to #1529, which this PR is based on. So worth reviewing that one first.