ARROW-14217: [Python][CI] Add support for python 3.10 #11316

kszucs · 2021-10-05T09:12:14Z

No description provided.

github-actions · 2021-10-05T09:16:22Z

https://issues.apache.org/jira/browse/ARROW-14217

kszucs · 2021-10-05T09:16:53Z

@github-actions crossbow submit cp310

github-actions · 2021-10-05T09:17:54Z

Revision: cd3da84

Submitted crossbow builds: ursacomputing/crossbow @ actions-883

Task	Status
wheel-macos-high-sierra-cp310-amd64
wheel-macos-mavericks-cp310-amd64
wheel-manylinux2010-cp310-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-windows-cp310-amd64

kszucs · 2021-10-05T09:18:15Z

@github-actions crossbow submit wheel-macos-big-sur-cp310*

github-actions · 2021-10-05T09:19:06Z

Revision: 583301f

Submitted crossbow builds: ursacomputing/crossbow @ actions-884

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2

kszucs · 2021-10-05T09:40:48Z

@github-actions crossbow submit cp310

github-actions · 2021-10-05T09:41:45Z

Revision: 583301f

Submitted crossbow builds: ursacomputing/crossbow @ actions-885

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-high-sierra-cp310-amd64
wheel-macos-mavericks-cp310-amd64
wheel-manylinux2010-cp310-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-windows-cp310-amd64

kszucs · 2021-10-05T10:45:00Z

@github-actions crossbow submit wheel-manylinuxcp310

github-actions · 2021-10-05T10:45:51Z

Revision: af016f3

Submitted crossbow builds: ursacomputing/crossbow @ actions-886

Task	Status
wheel-manylinux2010-cp310-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64

kszucs · 2021-10-05T12:28:41Z

@github-actions crossbow submit wheel-manylinuxcp310

github-actions · 2021-10-05T12:29:46Z

Revision: 93fc1b7

Submitted crossbow builds: ursacomputing/crossbow @ actions-887

Task	Status
wheel-manylinux2010-cp310-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64

pitrou · 2021-10-05T12:39:23Z

Why don't you start by adding a GHA entry for Python 3.10?

kszucs · 2021-10-05T14:25:46Z

Why don't you start by adding a GHA entry for Python 3.10?

You mean an additional conda based build?

kszucs · 2021-10-05T14:28:30Z

@github-actions crossbow submit wheel-manylinuxcp310

pitrou · 2021-10-05T14:40:10Z

You mean an additional conda based build?

Hmm, my bad, there's no conda package available.

kszucs · 2021-10-05T14:47:18Z

@github-actions crossbow submit wheel-manylinuxcp310

github-actions · 2021-10-05T15:16:27Z

Revision: ee6e3c6

Submitted crossbow builds: ursacomputing/crossbow @ actions-891

Task	Status
wheel-manylinux2010-cp310-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64

kszucs · 2021-10-05T16:37:00Z

The manylinux builds are working, but the testing docker images have not been released yet - matter of days I guess.

For the rest of the wheels we need to wait for the next numpy release.

kszucs · 2021-10-06T11:59:28Z

@github-actions crossbow submit wheel-manylinuxcp310

The python:3.10 images are available now https://hub.docker.com/_/python

github-actions · 2021-10-06T12:00:28Z

Revision: ee6e3c6

Submitted crossbow builds: ursacomputing/crossbow @ actions-896

Task	Status
wheel-manylinux2010-cp310-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64

github-actions · 2021-10-06T12:00:58Z

Revision: ee6e3c6

Submitted crossbow builds: ursacomputing/crossbow @ actions-897

Task	Status
wheel-manylinux2010-cp310-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64

github-actions · 2021-10-13T17:59:03Z

Revision: de75bb5

Submitted crossbow builds: ursacomputing/crossbow @ actions-939

Task	Status
wheel-manylinux2010-cp310-amd64
wheel-manylinux2010-cp36-amd64
wheel-manylinux2010-cp37-amd64
wheel-manylinux2010-cp38-amd64
wheel-manylinux2010-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp36-amd64
wheel-manylinux2014-cp36-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64

github-actions · 2021-10-13T17:59:32Z

Revision: de75bb5

Submitted crossbow builds: ursacomputing/crossbow @ actions-940

Task	Status
test-conda-python-3.10
test-conda-python-3.6
test-conda-python-3.6-pandas-0.23
test-conda-python-3.7
test-conda-python-3.7-dask-latest
test-conda-python-3.7-dask-master
test-conda-python-3.7-hdfs-2.9.2
test-conda-python-3.7-hdfs-3.2.1
test-conda-python-3.7-kartothek-latest
test-conda-python-3.7-kartothek-master
test-conda-python-3.7-pandas-0.24
test-conda-python-3.7-pandas-latest
test-conda-python-3.7-spark-branch-3.0
test-conda-python-3.8
test-conda-python-3.8-hypothesis
test-conda-python-3.8-pandas-latest
test-conda-python-3.8-pandas-nightly
test-conda-python-3.8-spark-master
test-conda-python-3.9
test-conda-python-3.9-pandas-master
test-debian-11-python-3
test-fedora-33-python-3
test-ubuntu-18.04-python-3

kszucs · 2021-10-13T18:12:39Z

Thanks @pitrou for the fix!

pitrou · 2021-10-13T18:14:00Z

@github-actions crossbow submit test-conda-python-3.10

github-actions · 2021-10-13T18:15:05Z

Revision: 90a0fd3

Submitted crossbow builds: ursacomputing/crossbow @ actions-941

Task	Status
test-conda-python-3.10

pitrou · 2021-10-13T18:37:21Z

"test-conda-python-3.10" will fail as long as Numpy and other packages are not available yet for Python 3.10 on conda-forge.

pitrou · 2021-10-13T18:40:11Z

@kszucs It seems vcpkg is building both debug and release versions of dependencies? See example

#12 1979.1 -- Downloading https://github.com/grpc/grpc/archive/44c40ac23023b7b3dd82744372c06817cc203898.tar.gz -> grpc-grpc-44c40ac23023b7b3dd82744372c06817cc203898.tar.gz...
#12 1979.1 -- Extracting source /opt/vcpkg/downloads/grpc-grpc-44c40ac23023b7b3dd82744372c06817cc203898.tar.gz
#12 1979.1 -- Applying patch 00001-fix-uwp.patch
#12 1979.1 -- Applying patch 00002-static-linking-in-linux.patch
#12 1979.1 -- Applying patch 00003-undef-base64-macro.patch
#12 1979.1 -- Applying patch 00004-link-gdi32-on-windows.patch
#12 1979.1 -- Applying patch 00005-fix-uwp-error.patch
#12 1979.1 -- Applying patch 00009-use-system-upb.patch
#12 1979.1 -- Applying patch 00010-add-feature-absl-sync.patch
#12 1979.1 -- Applying patch 00011-fix-csharp_plugin.patch
#12 1979.1 -- Applying patch snprintf.patch
#12 1979.1 -- Applying patch 00012-fix-use-cxx17.patch
#12 1979.1 -- Applying patch 00013-build-upbdefs.patch
#12 1979.1 -- Using source at /opt/vcpkg/buildtrees/grpc/src/17cc203898-db2679e7f2.clean
#12 1979.1 -- Configuring x64-linux-dbg
#12 1979.1 -- Configuring x64-linux-rel
#12 1979.1 -- Building x64-linux-dbg
#12 1979.1 -- Building x64-linux-rel
#12 1979.1 -- Installing: /opt/vcpkg/packages/grpc_x64-linux/share/grpc/copyright
#12 1979.1 -- Performing post-build validation
#12 1979.1 -- Performing post-build validation done

pitrou · 2021-10-13T19:03:00Z

For some reason, it looks like the 3.10 build doesn't use the Numpy binary wheels:
https://github.com/ursacomputing/crossbow/runs/3886034769?check_suite_focus=true#step:7:3186

kszucs · 2021-10-13T19:12:03Z

@kszucs It seems vcpkg is building both debug and release versions of dependencies? See example

Strange, the selected triplet file should restrict it to release builds.

kszucs · 2021-10-13T19:16:00Z

@kszucs It seems vcpkg is building both debug and release versions of dependencies? See example

Strange, the selected triplet file should restrict it to release builds.

Seems like that's not true for all of the packages.

kszucs · 2021-10-13T19:19:19Z

For some reason, it looks like the 3.10 build doesn't use the Numpy binary wheels:
https://github.com/ursacomputing/crossbow/runs/3886034769?check_suite_focus=true#step:7:3186

We may try to restrict that like in the macos builds.

kszucs · 2021-10-13T19:21:19Z

We may also need to bump the manylinux image tag because the interpreter path indicates cypthon version 3.10-rc2

kszucs · 2021-10-13T19:24:36Z

For some reason, it looks like the 3.10 build doesn't use the Numpy binary wheels:
https://github.com/ursacomputing/crossbow/runs/3886034769?check_suite_focus=true#step:7:3186

We may try to restrict that like in the macos builds.

Ahh, numpy only has wheels for manylinux_2014 and manylinux_2_17 and the build you referenced is manylinux2010

pitrou · 2021-10-14T08:30:48Z

We may also need to bump the manylinux image tag because the interpreter path indicates cypthon version 3.10-rc2

The wheel has the right tag apparently: "pyarrow-6.0.0.dev470-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl".
See https://github.com/ursacomputing/crossbow/runs/3886034769?check_suite_focus=true#step:7:6079

kszucs · 2021-10-14T11:32:50Z

I think it's worth waiting for:

the python 3.10 deployment in azure pipelines Add Python 3.10 (release) actions/runner-images#4226
which would make the numpy wheels available for windows and macos MAINT: Try making 3.10 wheels on Windows and OSX MacPython/numpy-wheels#135

pitrou · 2021-10-14T11:33:58Z

I'm not sure what waiting would bring here. We have a working fine to fix the tests already, and we can produce manylinux wheels. We can add the rest later.

kszucs · 2021-10-14T11:35:56Z

@github-actions crossbow submit wheel-manylinux*

kszucs · 2021-10-14T11:36:20Z

dev/tasks/python-wheels/github.linux.amd64.yml

@@ -47,7 +47,7 @@ jobs:
      {{ macros.github_upload_releases("arrow/python/repaired_wheels/*.whl")|indent }}
      {{ macros.github_upload_gemfury("arrow/python/repaired_wheels/*.whl")|indent }}

-      {% if arrow.branch == 'master' %}
+      {% if True or arrow.branch == 'master' %}


Need to remove before merge.

github-actions · 2021-10-14T11:37:00Z

Revision: 626987a

Submitted crossbow builds: ursacomputing/crossbow @ actions-954

Task	Status
wheel-manylinux2010-cp310-amd64
wheel-manylinux2010-cp36-amd64
wheel-manylinux2010-cp37-amd64
wheel-manylinux2010-cp38-amd64
wheel-manylinux2010-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp36-amd64
wheel-manylinux2014-cp36-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64

kszucs · 2021-10-14T11:38:45Z

I'm not sure what waiting would bring here. We have a working fine to fix the tests already, and we can produce manylinux wheels. We can add the rest later.

We can merge it now as well, but then we are going to have failing builds for the other platforms until numpy gains the new wheels. Ideally the only thing we'll need to do is to build and push the windows image for python 3.10, otherwise the builds should fix themselves.

Does that sound good to you?

pitrou · 2021-10-14T11:39:53Z

Yes, that sounds good to me.

kszucs

+1, merging on green

ursabot · 2021-10-18T10:23:17Z

Benchmark runs are scheduled for baseline = bea1701 and contender = 15f4e56. 15f4e56 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.13%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

kszucs changed the title ~~[Python][CI] Add support for python 3.10 [WIP]~~ ARROW-14217 [Python][CI] Add support for python 3.10 [WIP] Oct 5, 2021

kszucs changed the title ~~ARROW-14217 [Python][CI] Add support for python 3.10 [WIP]~~ ARROW-14217: [Python][CI] Add support for python 3.10 [WIP] Oct 5, 2021

apache deleted a comment from github-actions bot Oct 5, 2021

github-actions bot added the Component: Python label Oct 5, 2021

kantuni mentioned this pull request Oct 8, 2021

Streamlit fails to install w/ Python 3.10 (RC1 & RC2) - pyarrow failure to build streamlit/streamlit#3815

Closed

kszucs added 4 commits October 13, 2021 14:11

Add support for python 3.10

ae0e9ba

M1 wheels

284cb36

Quote python version

2c33bd7

Update windows dockerfile

4bf7b7c

Try to fix environment variable propagation

90a0fd3

pitrou changed the title ~~ARROW-14217: [Python][CI] Add support for python 3.10 [WIP]~~ ARROW-14217: [Python][CI] Add support for python 3.10 Oct 13, 2021

Bump manylinux docker tag and update numpy dependency for M1

626987a

kszucs commented Oct 14, 2021

View reviewed changes

Disable unconditional docker image push

d3e2278

kszucs commented Oct 15, 2021

View reviewed changes

kszucs closed this in 15f4e56 Oct 18, 2021

kou mentioned this pull request Jun 29, 2022

ARROW-16908: [Python][CI] Avoid installing wrong numpy version required for testing wheels #13449

Merged

asfimport mentioned this pull request Oct 18, 2021

[Python][CI] Add support for python 3.10 #29800

Closed

ARROW-14217: [Python][CI] Add support for python 3.10 #11316

ARROW-14217: [Python][CI] Add support for python 3.10 #11316

Conversation

kszucs commented Oct 5, 2021

github-actions bot commented Oct 5, 2021

kszucs commented Oct 5, 2021

github-actions bot commented Oct 5, 2021

kszucs commented Oct 5, 2021

github-actions bot commented Oct 5, 2021

kszucs commented Oct 5, 2021

github-actions bot commented Oct 5, 2021

kszucs commented Oct 5, 2021

github-actions bot commented Oct 5, 2021

kszucs commented Oct 5, 2021

github-actions bot commented Oct 5, 2021

pitrou commented Oct 5, 2021

kszucs commented Oct 5, 2021

kszucs commented Oct 5, 2021

pitrou commented Oct 5, 2021

kszucs commented Oct 5, 2021

github-actions bot commented Oct 5, 2021

kszucs commented Oct 5, 2021

kszucs commented Oct 6, 2021 • edited Loading

github-actions bot commented Oct 6, 2021

github-actions bot commented Oct 6, 2021

github-actions bot commented Oct 13, 2021

github-actions bot commented Oct 13, 2021

kszucs commented Oct 13, 2021

pitrou commented Oct 13, 2021

github-actions bot commented Oct 13, 2021

pitrou commented Oct 13, 2021

pitrou commented Oct 13, 2021

pitrou commented Oct 13, 2021

kszucs commented Oct 13, 2021

kszucs commented Oct 13, 2021

kszucs commented Oct 13, 2021

kszucs commented Oct 13, 2021

kszucs commented Oct 13, 2021

pitrou commented Oct 14, 2021

kszucs commented Oct 14, 2021

pitrou commented Oct 14, 2021

kszucs commented Oct 14, 2021

kszucs Oct 14, 2021

Choose a reason for hiding this comment

github-actions bot commented Oct 14, 2021

kszucs commented Oct 14, 2021

pitrou commented Oct 14, 2021

kszucs left a comment

Choose a reason for hiding this comment

ursabot commented Oct 18, 2021 • edited Loading

kszucs commented Oct 6, 2021 •

edited

Loading

ursabot commented Oct 18, 2021 •

edited

Loading