fix: ensure the v2 file writer does not mix up the order of a column

[westonpace]

Sometimes the v2 writer will split a batch into multiple pages. When it does this it encodes those pages in parallel. It is possible those encoding tasks finish out of order and the writer was then writing the pages out of order. This mean the order of one column could get out of sync with the order of another column.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.05%. Comparing base (f30a679) to head (60cfb22).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2836   +/-   ##
=======================================
  Coverage   78.04%   78.05%           
=======================================
  Files         229      229           
  Lines       70299    70299           
  Branches    70299    70299           
=======================================
+ Hits        54866    54871    +5     
+ Misses      12339    12332    -7     
- Partials     3094     3096    +2     

Flag	Coverage Δ
unittests	78.05% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[broccoliSpicy]

is there any reason we don't like adding a page_idx field in EncodedPage then sort it in write_pages?

one benefit could be that we can still write pages in different columns parallelly

[westonpace]

@broccoliSpicy that's pretty much what FuturesOrdered is doing. It will still run all the encode tasks in parallel, it just sorts the results when it collects them before the write to disk.