feat: merge-insert supports inserting subset of columns

[wjones127]

In #2639 we added support for updating subcolumns. In #3041 we added support for inserting subcolumns. This PR adds support for upserting them (or doing insert-if-not-exists).

Closes #2904

Example

import pyarrow as pa
import lance

table = pa.table({
    "id": range(3),
    "a": [1.0, 2.0, 3.0],
    "c": ["x", "x", "x"]
})
dataset = lance.write_dataset(table, "example")

# Upsert: when_matched_update_all + when_not_matched_insert_all
new_data = pa.table({
    "id": [2, 3],
    "c": ["y", "y"]
})
(
    dataset
    .merge_insert(on="id")
    .when_matched_update_all()
    .when_not_matched_insert_all()
    .execute(new_data)
)
dataset.to_table().to_pandas()

   id    a  c
0   0  1.0  x
1   1  2.0  x
2   2  3.0  y
3   3  NaN  y

# Insert-if-not-exists: when_not_matched_insert_all
new_data = pa.table({
    "id": [3, 4],
    "c": ["z", "z"]
})
(
    dataset
    .merge_insert(on="id")
    .when_not_matched_insert_all()
    .execute(new_data)
)
dataset.to_table().to_pandas()

   id    a  c
0   0  1.0  x
1   1  2.0  x
2   2  3.0  y
3   3  NaN  y
4   4  NaN  z

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 83.83838% with 16 lines in your changes missing coverage. Please review.

Project coverage is 78.91%. Comparing base (f2906cf) to head (e7a79cf).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/write/merge_insert.rs	83.83%	9 Missing and 7 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3100      +/-   ##
==========================================
- Coverage   78.97%   78.91%   -0.06%     
==========================================
  Files         246      246              
  Lines       86313    86560     +247     
  Branches    86313    86560     +247     
==========================================
+ Hits        68162    68311     +149     
- Misses      15328    15421      +93     
- Partials     2823     2828       +5     

Flag	Coverage Δ
unittests	78.91% <83.83%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[westonpace]

Nice work!

[westonpace]

Not needed for this PR but we should add a drop columns method to schema.