Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyArrow version requirement #55

Closed
YS-L opened this issue Oct 5, 2022 · 7 comments
Closed

PyArrow version requirement #55

YS-L opened this issue Oct 5, 2022 · 7 comments

Comments

@YS-L
Copy link

YS-L commented Oct 5, 2022

Hello! Is the version requirement on the pyarrow package strict?

pyarrow = "^9.0.0"

We would like to use databricks-sql-python in a project that depends on a lower version of pyarrow (8.0.0). We ran into conflicting version issues when using package management tools such as poetry. In our tests it seems databricks-sql-python can work with older versions of pyarrow such as 8.0.0. It would be great if databricks-sql-python can specify a wider range of pyarrow versions for better compatibility.

@Tom-Newton
Copy link
Contributor

I have a similar issue but personally I'm interested in using pyarrow 10.0.0. It would be great if this was allowed.

@susodapop
Copy link
Contributor

@Tom-Newton pyarrow 10.0.1 is now supported as our v2.2.1 release.

@YS-L I'll look into reducing our pyarrow specificaiton. I think we landed on version 9 because it was the latest at the time we wrote the connector.

@Tom-Newton
Copy link
Contributor

Tom-Newton commented Dec 8, 2022

Thanks @susodapop 🙂 . With 2.2.1 my pyarrow problem is solved but now numpy is an issue. We are on numpy 1.22.4 with python 3.8. Is there are reason that the numpy requirement needs to be so strict?

We are currently using databricks-sql-connector==2.0.4 with numpy==1.22.4 and we haven't noticed any issues.

@susodapop
Copy link
Contributor

I want to make this work for you but I also can't reproduce your installation problem. For your info: note that our specific requirements were dictated by Poetry so we haven't deliberately excluded anyone and thus far our tests run automatically on Python 3.7-3.11 without issues.

You're welcome to open a pull request that modifies pyproject.toml and poetry.lock to meet your needs. As long as it passes our build checks I'm happy to merge it. But this isn't something we're prepared to do on our own because nothing is broken for us.

@Tom-Newton
Copy link
Contributor

I created a PR #74. It needs a review and for someone to approve running the build checks.

@Tom-Newton
Copy link
Contributor

I created a PR #74. It needs a review and for someone to approve running the build checks.

@susodapop please could you approve running the github workflows
image

@susodapop
Copy link
Contributor

Done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants