Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

arrow 14.0.0 with new setup (more granular libraries; no more legacy outputs) #1201

Merged
merged 16 commits into from
Nov 7, 2023
Merged

arrow 14.0.0 with new setup (more granular libraries; no more legacy outputs) #1201

merged 16 commits into from
Nov 7, 2023

Conversation

h-vetinari
Copy link
Member

@h-vetinari h-vetinari commented Oct 14, 2023 •
edited
Loading

Combines #1170 with a slimmed-down version of #1175

Closes #1035
Closes #1170
Closes #1175
Closes #1218

@conda-forge-webservices
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

  • It looks like the 'libarrow-all' output doesn't have any tests.

@h-vetinari h-vetinari force-pushed the 14_split branch 4 times, most recently from 3685764 to efb4224 Compare October 16, 2023 02:52
@conda-forge-webservices
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge conda-forge deleted a comment from conda-forge-webservices bot Oct 16, 2023
@h-vetinari h-vetinari force-pushed the 14_split branch 3 times, most recently from 6170f32 to 6173572 Compare October 16, 2023 04:02
@conda-forge conda-forge deleted a comment from conda-forge-webservices bot Oct 16, 2023
@h-vetinari h-vetinari force-pushed the 14_split branch 3 times, most recently from 9a36abb to 74cecc8 Compare October 16, 2023 12:41
@h-vetinari h-vetinari mentioned this pull request Oct 16, 2023
Closed
10 tasks
@h-vetinari h-vetinari mentioned this pull request Nov 1, 2023
Closed
3 tasks
@h-vetinari h-vetinari changed the title WIP: arrow 14 with component split arrow 14.0.0 with new setup (more granular libraries; no more legacy outputs) Nov 1, 2023
@h-vetinari h-vetinari marked this pull request as ready for review November 1, 2023 07:53
xhochy
xhochy approved these changes Nov 7, 2023
View reviewed changes
Copy link
Member

@xhochy xhochy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are probably ways to speed up the build a bit but from the content-perspective, this PR looks good to me.

h-vetinari and others added 16 commits November 7, 2023 21:55
@h-vetinari
remove compat outputs
87d85ee
@h-vetinari
add llvm host-dep for libarrow
0c00a15
@h-vetinari
use newer CUDAToolkit_ROOT variable for CMake
6032175
@h-vetinari
mark test skip for unsupported characters on win as avoidable
65164c1
@h-vetinari
add remaining test skips
164b4fe
@raulcd @h-vetinari
Split libarrow into components
3b0fb71 
Reduce libarrow to its core components, and distinguish:
    libarrow-{acero,dataset,flight,flight-sql,gandiva,substrait}
    libparquet

Introduce libarrow-all to represent "all of libarrow",
and build pyarrow on top

Co-Authored-By: H. Vetinari <h.vetinari@gmx.com>
@h-vetinari
ensure we get the right llvm version
59aac8b
@h-vetinari
avoid picking up gdb symlink multiple times
369e9e0
@regro-cf-autotick-bot @h-vetinari
updated v14.0.0
959f77f
@h-vetinari
rebase patches
8f6ff2c
@h-vetinari
remove obsolete test skip
6f76392
@h-vetinari
Revert "cython <3"
6d7d316 
This reverts commit 1a627c0.
@h-vetinari
Revert "Add test pinnings for fsspec and s3fs less than 2023.10.0."
eb4268c 
This reverts commit ea92639.
@h-vetinari
backport patch for (test) compat with newer fsspec
2cf4102
@h-vetinari
reintroduce patch for compat with newer minio
31b375f
@h-vetinari
MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.27.1, and co…
a19a47b 
…nda-forge-pinning 2023.11.07.08.59.34
@h-vetinari h-vetinari added the automerge Merge the PR when CI passes label Nov 7, 2023
@h-vetinari
Copy link
Member Author

Thanks for the review! :)

@github-actions github-actions bot merged commit 3eaea8a into conda-forge:main Nov 7, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 7, 2023

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

  • linter: passed
  • azure: passed

Thus the PR was passing and merged! Have a great day!

@bdice bdice mentioned this pull request Nov 7, 2023
Closed
@jorisvandenbossche
Copy link
Member

jorisvandenbossche commented Nov 9, 2023 •
edited
Loading

@h-vetinari thanks a lot for finishing this splitting! Based on a quick test, this seems to be working fine

As a follow-up, should we add a pyarrow-base or pyarrow-core (I think both suffixes are being used) that only depends on libarrow?

@raulcd
Copy link
Member

raulcd commented Nov 9, 2023

I was planning on trying to work on this during the next couple of weeks now that this has been merged and I have released 14.0.1.

@h-vetinari h-vetinari deleted the 14_split branch November 9, 2023 22:36
@h-vetinari
Copy link
Member Author

As a follow-up, should we add a pyarrow-base or pyarrow-core (I think both suffixes are being used) that only depends on libarrow?

Based on discussion in apache/arrow#38536 and the work towards that which went into arrow 14 already, I think we should definitely try!

I was planning on trying to work on this during the next couple of weeks now that this has been merged and I have released 14.0.1.

Great!

@raulcd raulcd mentioned this pull request Nov 27, 2023
Closed
5 tasks
@jjerphan jjerphan mentioned this pull request Jan 25, 2024
Closed
@h-vetinari h-vetinari mentioned this pull request May 6, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xhochy xhochy xhochy approved these changes

@cpcloud cpcloud Awaiting requested review from cpcloud cpcloud is a code owner

@jakirkham jakirkham Awaiting requested review from jakirkham jakirkham is a code owner

@jreback jreback Awaiting requested review from jreback jreback is a code owner

@kou kou Awaiting requested review from kou kou is a code owner

@kszucs kszucs Awaiting requested review from kszucs kszucs is a code owner

@leifwalsh leifwalsh Awaiting requested review from leifwalsh leifwalsh is a code owner

@nealrichardson nealrichardson Awaiting requested review from nealrichardson nealrichardson is a code owner

@pcmoritz pcmoritz Awaiting requested review from pcmoritz pcmoritz is a code owner

@pearu pearu Awaiting requested review from pearu pearu is a code owner

@pitrou pitrou Awaiting requested review from pitrou pitrou is a code owner

@robertnishihara robertnishihara Awaiting requested review from robertnishihara robertnishihara is a code owner

@siddharthteotia siddharthteotia Awaiting requested review from siddharthteotia siddharthteotia is a code owner

@wesm wesm Awaiting requested review from wesm wesm is a code owner

Assignees
No one assigned
Labels
automerge Merge the PR when CI passes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Package some form of "minimal" libarrow?
5 participants
@h-vetinari @jorisvandenbossche @raulcd @xhochy @regro-cf-autotick-bot