Help pip resolver to make better decisions on Pyarrow version #25697
Conversation
The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it.
|
In case we decided to do 2.3.4 - we will have to cherry-pick that one - marked it as 2.3.4 |
|
A colleague in Astronomer just identified the root cause to this. databricks-sql-python, a dependency of the databricks provider, does not support 3.10. It also does not support pyarrow>=6, so pip is correct to report a resolution is impossible, although that conflict is the less important issue of the two and thus sort of misleading. But in any case, this means the latest constraint files will probably maybe have issues with the databricks extra, but I’m willing to bet on there’s nothing noticable unless someone complains on the issue tracker. |
That must be it, indeed. https://pypi.org/project/databricks-sql-connector/2.0.3/ was released 7 days ago so that adds-up. @alexott - maybe you could ping the right people at Databricks and help to make the connector 3.10 compliant ? |
|
Ok, let me point team to it |
|
Hey team, |
…apache#25697)" This reverts commit 1b84048.
Woa.. That was QUICK @susodapop (And @alexott). PR running here: #25777 - if it succeeds we are good. |
|
Yep. looks good @alexott @susodapop :) . All the images got built so it looks like it is going to work :) |
|
(but it does not actually mean the 2.0.4 upgrade was bad :) @susodapop @alexott :)). |
…#25697) The pip resolver in `eager upgrade` mode on Python 3.10 decides to downgrade Pyarrow to 5.0.0 which triggers numpy not being compatible with 3.10 - even if Pyarrow 6.* is perfectly fine. Adding the limit to eager upgrade directly helps the resolver to make better decisions on it. (cherry picked from commit 1b84048)
The pip resolver in
eager upgrademode on Python 3.10 decidesto downgrade Pyarrow to 5.0.0 which triggers numpy not being
compatible with 3.10 - even if Pyarrow 6.* is perfectly fine.
Adding the limit to eager upgrade directly helps the resolver to
make better decisions on it.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.