Parquet writer should not write any min/max data to ColumnIndex when all values are null #6315
Labels
enhancement
Any new improvement worthy of a entry in the changelog
parquet
Changes to the parquet crate
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
While investigating #6310 I noticed that when a page contains only null values, min and max for that page will be set to
vec![0; 1]
. Because of this, a byte array of length 1 will be written. Instead an empty vec should be passed so spurious min/max values are not produced.Describe the solution you'd like
Rather than
vec![0; 1]
,vec![]
should be passed toColumnIndexBuilder::append
. This will reduce the column index size a tiny bit.Describe alternatives you've considered
It could be the current behavior is desired, so no change would be necessary.
Additional context
The current behavior isn't a bug, but is a little wasteful.
The text was updated successfully, but these errors were encountered: