Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update for latest geoarrow #56

Closed
kylebarron opened this issue Oct 2, 2024 · 2 comments
Closed

Update for latest geoarrow #56

kylebarron opened this issue Oct 2, 2024 · 2 comments

Comments

@kylebarron
Copy link
Contributor

I'd love to update h3arrow for the latest geoarrow-rs, and potentially use h3arrow as an (optional) dependency from Lonboard, a Python library for large geospatial vector data visualization.

Some changes:

  • As of this week (unreleased) geoarrow-rs does not have an O: OffsetSizeTrait generic on the PolygonArray: Remove "large" arrays and scalars with i64 offsets geoarrow/geoarrow-rs#803. In order to overflow the offsets of an int32, you'd need to have more than 2^32 coordinates in a single contiguous array, which would take up 32GB alone. Removing this generic makes it easier to move faster in geoarrow-rs and took out a ton of complexity.
  • On the contrary, I think supporting multiple dimensions in geoarrow-rs is necessary complexity. So arrays now have a const D: usize generic for the physical dimension. So now when you construct arrays you'd have PolygonArray<2>.
  • I started to implement a pyo3-geoarrow Rust helper module to make it easier to handle zero-copy data interop from Python, but that's not yet fully standardized.

Let me know if I can help or give pointers to any of this.

@nmandery
Copy link
Owner

nmandery commented Oct 4, 2024

Great. I did not know about lonboard - looks good. Do you want to use the h3arrow rust library as a optional dependency or h3ronpy? I just moved h3arrow into the h3ronpy repo and planned to not push it to crates.io anymore, as there are no other users. In case you need it, I can keep publishing it.

I started migrating to arrow 52 and geoarrow 0.3, but got side-tracked during working on this - so I did not finish it. I will try to push it ahead. Thanks for the migration hints 👍

nmandery added a commit that referenced this issue Oct 4, 2024
@kylebarron
Copy link
Contributor Author

Do you want to use the h3arrow rust library as a optional dependency or h3ronpy?

Sorry I wasn't clear. Ideally I'd use h3ronpy and not make my own Python bindings (lonboard doesn't use Rust itself, only via dependencies, and thus doesn't have a build system in place).

nmandery added a commit that referenced this issue Oct 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants