[WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 #45551
Conversation
|
Thank you, @panbingkun . Please let me know when this PR is ready. |
|
I backported #42897 to branch-3.5 and branch-3.4. Could you rebase this PR to the |
Okay, I will verify it first today. |
|
I temporarily removed some unrelated tests to make it faster. |
| @@ -382,17 +219,10 @@ jobs: | |||
| with: | |||
| fetch-depth: 0 | |||
| repository: apache/spark | |||
| ref: ${{ inputs.branch }} | |||
| ref: ${{ matrix.branch }} | |||
| @@ -438,11 +268,21 @@ jobs: | |||
| curl -s https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > miniconda.sh | |||
| bash miniconda.sh -b -p $HOME/miniconda | |||
| rm miniconda.sh | |||
| - name: Install Python test dependencies for branch-3.4 | |||
There was a problem hiding this comment.
In order to pass the pyspark testing of branch-3.4 and branch-3.5.
| # Run the tests. | ||
| - name: Run tests | ||
| env: ${{ fromJSON(inputs.envs) }} | ||
| shell: 'script -q -e -c "bash {0}"' | ||
| run: | | ||
| export SCALA_PROFILE="scala2.13" | ||
| unset GITHUB_ACTIONS |
There was a problem hiding this comment.
hack git compare
.github/workflows/build_and_test.yml
Outdated
| run: | | ||
| # SPARK-45212: Copy from https://github.com/apache/spark/blob/555c8def51e5951c7bf5165a332795e9e330ec9d/.github/workflows/build_and_test.yml#L631-L638 | ||
| # Should delete this section after SPARK 3.5 EOL. | ||
| python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme 'mypy==0.982' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' numpydoc 'jinja2<3.0.0' 'black==22.6.0' | ||
| python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme 'mypy==0.982' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' 'numpy==1.25.1' 'pyarrow==12.0.1' numpydoc 'jinja2<3.0.0' 'black==22.6.0' 'pandas<=2.0.3' 'matplotlib==3.7.2' 'torch==2.0.1' 'torchvision==0.15.2' |
There was a problem hiding this comment.
For fixing as
/usr/local/lib/python3.9/dist-packages/torch/_dynamo/mutation_guard.py:1: error: disable_error_code: Invalid error code(s): method-assign [misc]
/usr/local/lib/python3.9/dist-packages/torch/_dynamo/eval_frame.py:1: error: disable_error_code: Invalid error code(s): method-assign [misc]
/usr/local/lib/python3.9/dist-packages/torch/_dynamo/debug_utils.py:1: error: disable_error_code: Invalid error code(s): method-assign [misc]
python/pyspark/pandas/plot/matplotlib.py:23: error: Module "matplotlib.axes._base" has no attribute "_process_plot_format" [attr-defined]
Found 4 errors in 4 files (checked 688 source files)
|
This PR is basically successful, and I will slightly organize it and submit it as a separate PR. This PR allows me to keep it as a reference for future testing of the branch. |
What changes were proposed in this pull request?
The pr aims to fix
PySpark Test scheduled jobsforbranch-3.4&branch-3.5.Why are the changes needed?
Fix github jobs.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.
Was this patch authored or co-authored using generative AI tooling?
No.