feat: reconnect block compression

[westonpace]

This allows block compression (zstd) in the narrow case of using it for binary data. A more generalized approach to block compression can be handled as part of 2.1.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 81.08108% with 7 lines in your changes missing coverage. Please review.

Project coverage is 77.84%. Comparing base (9c361fe) to head (0f80146).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...st/lance-encoding/src/encodings/physical/binary.rs	78.26%	3 Missing and 2 partials ⚠️
rust/lance-encoding/src/encoder.rs	85.71%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2878      +/-   ##
==========================================
- Coverage   77.98%   77.84%   -0.15%     
==========================================
  Files         231      231              
  Lines       70643    70199     -444     
  Branches    70643    70199     -444     
==========================================
- Hits        55090    54643     -447     
- Misses      12424    12685     +261     
+ Partials     3129     2871     -258     

Flag	Coverage Δ
unittests	77.84% <81.08%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wjones127]

We should probably expose the compression level. Then users could tune the compression ratio to compression speed tradeoff.

[wjones127]

clippy

Suggested change

	self.compression_scheme.clone(),
	self.compression_scheme,

[westonpace]

We should probably expose the compression level.

@wjones127 ah, now we run into the question of how we should allow this kind of advanced configuration

We could use a separate metadata key lance-encoding:compression-level or we could set the lance-encoding:compression key to a non-trivial value (e.g. { "lance-encoding:compression": "zstd-3" }). (or there are probably a lot of other things we can do).

In general, I don't like exposing too much configuration to the user (which seems hypocritical to say since I think I've added half a dozen environment variables in the last month) without good reason. Why would a user choose something other than the default compression level?

[niyue]

We should probably expose the compression level

We could use a separate metadata key lance-encoding:compression-level or we could set the lance-encoding:compression key to a non-trivial value (e.g. { "lance-encoding:compression": "zstd-3" })

Why would a user choose something other than the default compression level?

@wjones127 @westonpace
I would like to know whether exposing the compression level as a configurable option is still something being considered. If this is considered, I would be happy to submit a PR for it.

Zstd offers 22 compression levels, each yielding different compression ratios. The compressed data size can vary by up to 50% between levels 1 and 22, depending on data distribution (e.g., [1]). Some databases, like AWS Athena, allow users to specify the Zstd compression level via DDL [2]. Similarly, it would be beneficial if Lance provided the ability to customize compression settings, enabling users to balance time and space efficiency based on their use cases. Thanks.

[1] https://www.reddit.com/r/compression/comments/18e524n/zstd_compression_ratios_by_level/
[2] https://docs.aws.amazon.com/athena/latest/ug/compression-support-zstd-levels.html

[wjones127]

I'm for it, if we can find a sensible place to put it.

[westonpace]

I'm ok with exposing it. I'd prefer a entry in the field metadata like lance-encoding:compression-level.