feat: more flexible and sophisticated handling of non-null constraints

[westonpace]

Previously, we used batch schema to check if incoming data was compatible with the dataset. This has a few problems:

• You couldn't add a non-nullable batch to a nullable dataset even though this should be perfectly valid
• You couldn't add a nullable batch to a non-nullable dataset even when the batch didn't contain nulls
• It's possible (and with pyarrow, easy) to create a batch that has nulls but the fields are marked non-null, this led to confusing errors.

This PR resolves those issues by checking the null count of the actual arrays instead of relying on the schema assigned to the data.

Closes #1936

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 88.31169% with 9 lines in your changes missing coverage. Please review.

Project coverage is 78.39%. Comparing base (f17d88d) to head (36b13dc).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance-file/src/writer.rs	76.19%	1 Missing and 4 partials ⚠️
rust/lance-file/src/v2/writer.rs	76.47%	1 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2467      +/-   ##
==========================================
+ Coverage   78.24%   78.39%   +0.14%     
==========================================
  Files         240      240              
  Lines       77284    78593    +1309     
  Branches    77284    78593    +1309     
==========================================
+ Hits        60470    61611    +1141     
- Misses      13696    13850     +154     
- Partials     3118     3132      +14     

Flag	Coverage Δ
unittests	78.39% <88.31%> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wjones127]

Nice work! I appreciate all the tests.

[wjones127]

Not necessary, but would be cool to have a test showing you can use alter_columns to change the nullability, and that inserting nulls would then work without an error.

[westonpace]

I'll try and add one later today but want to merge this now to clear my plate 🤷