Fail to run tests for peg generator under Python 3.11.4

[ghost]

Bug report

Bug description:

I am running Python 3.11.4 on Ubuntu 22.04.3 and experienced the following test failure when trying to compile Python 3.11.5. Every sub-test for the peg generator failed with the same error status:

ModuleNotFoundError: No module named 'distutils.tests'

I followed the procedure described below:

 ./configure CFLAGS='-DPYMALLOC_DEBUG' --with-pydebug
 make
 make test

I have been unable to attach the log file as it is too big (2000+) lines but it is stored on my system and when I can find a way it can be supplied.

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Some notes:

• distutils.test does not appear to be in the GitHub repository
• Optimized Python builds with configure --enable-optimizations work

[ghost]

I checked and the file distutils.test was not found anywhere on my system

[iritkatriel]

distutils was removed in python 3.12. Could it be that you are running the 3.11 test suite with a 3.12 python?

[smontanaro]

This seems stale, but I ran into it today. (I built and tested 3.11 alongside other versions on a regular basis.) I am just executing ./configure with no command line args. I get the peg failures, like this:

Traceback (most recent call last):
  File "/Users/skip/src/python/py3.11/Lib/test/test_peg_generator/test_c_parser.py", line 225, in test_left_recursion
    self.run_test(grammar_source, test_source)
  File "/Users/skip/src/python/py3.11/Lib/test/test_peg_generator/test_c_parser.py", line 115, in run_test
    self.build_extension(grammar_source)
  File "/Users/skip/src/python/py3.11/Lib/test/test_peg_generator/test_c_parser.py", line 112, in build_extension
    generate_parser_c_extension(grammar, Path('.'), library_dir=self.library_dir)
  File "/Users/skip/src/python/py3.11/Tools/peg_generator/pegen/testutil.py", line 105, in generate_parser_c_extension
    compile_c_extension(
  File "/Users/skip/src/python/py3.11/Tools/peg_generator/pegen/build.py", line 54, in compile_c_extension
    from distutils.tests.support import fixup_build_ext  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'distutils.tests'

This is in my 3.11 repo with no changes. I'm not sure why distutils.tests isn't found. It's available at the interpreter prompt:

% ./python.exe 
Python 3.11.8+ (heads/3.11:d83b4c570c, Mar 14 2024, 18:34:44) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.tests
<stdin>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

The full failing import also succeeds at the interpreter prompt:

% ./python.exe
Python 3.11.8+ (heads/3.11:d83b4c570c, Mar 14 2024, 18:34:44) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.tests.support import fixup_build_ext
<stdin>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

It's almost as if sys.modules["distutils"] doesn't refer to the (deprecated) module we're expecting.

[zware]

It's almost as if sys.modules["distutils"] doesn't refer to the (deprecated) module we're expecting.

That sounds a lot like setuptools shenanigans.

[smontanaro]

The question would then be what changed with setuptools? Looking at the 3.11 commit log, the last change I see which mentions that package is from July 2022. I backed up to the commit just before that, then ran:

git clean -fdx
./configure
make -j
./python.exe -E  ./Tools/scripts/run_tests.py test_peg_generator

and it still failed. It wouldn't seem to be anything obvious with a recent version of setuptools.

My day-to-day box is a MacBook Pro M1. I tried the same experiment on my old Dell laptop (running up-to-date XUbuntu 22.04). The test succeeded. It's not at all obvious to me why this test might be platform- or architecture-dependent. That I can get it to fail so far back in time (when it has routinely been succeeding for me until the past day or two) suggests maybe a change in the external tool chain is (partly?) to blame.

[lysnikolaou]

Indeed, setuptools is the culprit here. See #91169 and pypa/setuptools#3007 for more info.

Running the tests with a freshly built 3.11 without setuptools succeeds.It's only after running make altinstall that it starts to fail, cause setuptools has been installed.

There's always the work-around of setting the SETUPTOOLS_USE_DISTUTILS=stdlib environment variable:

❯ SETUPTOOLS_USE_DISTUTILS=stdlib ./python.exe -m test test_peg_generator   
Using random seed: 144407516
0:00:00 load avg: 2.25 Run 1 test sequentially
0:00:00 load avg: 2.25 [1/1] test_peg_generator
/Users/lysnikolaou/repos/python/cpython-versions/3.11-dev/Lib/test/support/__init__.py:1687: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils import ccompiler, sysconfig, spawn, errors
/Users/lysnikolaou/repos/python/cpython-versions/3.11-dev/Lib/test/support/__init__.py:1687: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
  from distutils import ccompiler, sysconfig, spawn, errors

== Tests result: SUCCESS ==

1 test OK.

Total duration: 13.7 sec
Total tests: run=91
Total test files: run=1/1
Result: SUCCESS

I'm not sure if we want to take any further action here, and if we do, it's probably something that should be fixed on the setuptools side.

[smontanaro]

Ah, that rings a bell! I only recently installed my 3.11 build because I wanted to tox run in the cherry-picker repo. Previously, I'd just built the interpreter and run the test suite without installing...

[smontanaro]

I removed my locally installed Python 3.11 and tried again. The test suite ran without error.

[thehedhly]

The "test_peg_generator" test is also failing for Python-3.12.4.
Source: https://www.python.org/ftp/python/3.12.4/Python-3.12.4.tgz
Host:

Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

arm64

Logs:

0:08:28 load avg: 4.81 [488/489/1] test_hashlib passed (1 min 8 sec)
0:08:32 load avg: 4.81 [489/489/1] test.test_gdb.test_pretty_print passed (31.8 sec)

== Tests result: FAILURE ==

33 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_bz2 test_ctypes
    test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_gzip
    test_idle test_ioctl test_kqueue test_launcher test_lzma
    test_msilib test_readline test_smtpnet test_sqlite3 test_ssl
    test_stable_abi_ctypes test_startfile test_tcl test_tix
    test_tkinter test_ttk test_ttk_textonly test_turtle test_winapi
    test_winconsoleio test_winreg test_wmi test_zlib test_zoneinfo

3 tests skipped (resource denied):
    test_ossaudiodev test_winsound test_zipfile64

1 test failed:
    test_peg_generator

452 tests OK.

0:08:32 load avg: 4.81 Re-running 1 failed tests in verbose mode in subprocesses
0:08:32 load avg: 4.81 Run 1 test in parallel using 1 worker process
0:08:33 load avg: 4.81 [1/1/1] test_peg_generator failed (1 error)
Re-running test_peg_generator in verbose mode (matching: TestCParser)

Run: /home/foo/Python-3.12.4/python -X dev -m venv venv
Error: Command '['/home/foo/Python-3.12.4/build/test_python_175030æ/tempcwd/venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
setUpClass (test.test_peg_generator.test_c_parser.TestCParser) ... ERROR

======================================================================
ERROR: setUpClass (test.test_peg_generator.test_c_parser.TestCParser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/foo/Python-3.12.4/Lib/test/test_peg_generator/test_c_parser.py", line 102, in setUpClass
    python_exe = stack.enter_context(support.setup_venv_with_pip_setuptools_wheel("venv"))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/foo/Python-3.12.4/Lib/contextlib.py", line 526, in enter_context
    result = _enter(cm)
             ^^^^^^^^^^
  File "/home/foo/Python-3.12.4/Lib/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/foo/Python-3.12.4/Lib/test/support/__init__.py", line 2215, in setup_venv_with_pip_setuptools_wheel
    subprocess.run(cmd, check=True)
  File "/home/foo/Python-3.12.4/Lib/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/foo/Python-3.12.4/python', '-X', 'dev', '-m', 'venv', 'venv']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 0 tests in 0.178s

FAILED (errors=1)
test test_peg_generator failed
1 test failed again:
    test_peg_generator

== Tests result: FAILURE then FAILURE ==

33 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_bz2 test_ctypes
    test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_gzip
    test_idle test_ioctl test_kqueue test_launcher test_lzma
    test_msilib test_readline test_smtpnet test_sqlite3 test_ssl
    test_stable_abi_ctypes test_startfile test_tcl test_tix
    test_tkinter test_ttk test_ttk_textonly test_turtle test_winapi
    test_winconsoleio test_winreg test_wmi test_zlib test_zoneinfo

3 tests skipped (resource denied):
    test_ossaudiodev test_winsound test_zipfile64

1 re-run test:
    test_peg_generator

1 test failed:
    test_peg_generator

452 tests OK.

Total duration: 8 min 33 sec
Total tests: run=39,939 skipped=1,883
Total test files: run=487/489 failed=1 skipped=33 resource_denied=3 rerun=1
Result: FAILURE then FAILURE
make: *** [Makefile:1956: test] Error 2

[picnixz]

@thehedhly Can you try with the latest 3.12.X?

[picnixz]

Closince as cannot reproduce since the 3.11 issue was solved. For those having the issue with 3.12, please open a separate issue as the culprit is likely something else.