Cargo's resolution of major versions is majorly annoying.

[rtyler]

The release of 0.6.1 by @zachschuermann gave us the real-world opportunity to see how well version ranges would work for arrow dependencies of kernel. The status quo at the time of this writing is that the latest arrow is 54 but the latest datafusion crate (44) depends on 53. 🕵️

I posted this topic on the rust-lang forum, but in short, the following Cargo.toml will likely never do what we want it to 😢

[package]
name = "version-ranges-make-me-sad"
version = "0.1.0"
edition = "2021"

[dependencies]
arrow = "53.3.0"
# Specifies arrow >=53, <55
delta_kernel = { version = "0.6.1", features = ["default-engine"]}

What I would love to happen, and what in a just and sane world I would expect to happen is that the cargo resolver would chose arrow 53.3.0 since the package expressly requests 53.3.0 and the range used by delta_kernel 0.6.1 accommodates this version as well.

Instead what happens is that cargo gets silly goofy and gives the package 53.3.0 but then compiles delta_kernel with 54.0.0 and then the poor unfortunate user (me) is hit with a lot of ABI compilation failures because a Schema in 53 is different than a Schema in 54 (or some such nonsense).

Regrettably I think this leaves our only remaining options to be:

• Pick a one true version for arrow (:rage:)
• Incorporate feature flags such as arrow_53, arrow_54, etc and keep a rolling window for that.

I dislike both options but the second I have seen done conventionally in other crates and I believe it to be our only option to not require delta_kernel to be on a painful release treadmill chasing after arrow.

[zachschuermann]

thanks @rtyler for raising this. unfortunate that we don't get the desired behavior with our version ranges :(

I would vote for just separate arrow version feature flags and we can support some finite range of versions.

wdyt @nicklan? sounds like we will do a 0.7.0 with some new approach?

[scovich]

side rant: Rust as a language is admirably flexible and accommodating of real-life user needs, but rough edges in cargo somehow seem impossible to make any progress on (see also the inanity of cargo using stderr internally for communication). I don't understand it.