Propagate --use-pep517 to build-system requirements

[fgregg]

Description

when installing a package that has build-system requirements, the --use-pep517 argument is not propagated to the building of those requirements.

for example https://github.com/zopefoundation/BTrees/blob/master/pyproject.toml

Expected behavior

--use-pep517 will propagate

pip version

pip 22.3

Python version

python 3.11

OS

OSX

How to Reproduce

pip install BTrees on Mac OSX with python 3.11

Output

Collecting BTrees>=4.1.4
  Downloading BTrees-4.10.1.tar.gz (195 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 195.8/195.8 kB 12.5 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [77 lines of output]
      Collecting setuptools
        Using cached setuptools-65.5.0-py3-none-any.whl (1.2 MB)
      Collecting wheel
        Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
      Collecting persistent
        Downloading persistent-4.9.1.tar.gz (125 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.3/125.3 kB 3.5 MB/s eta 0:00:00
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'done'
      Collecting zope.interface
        Downloading zope.interface-5.5.0.tar.gz (253 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 253.3/253.3 kB 10.6 MB/s eta 0:00:00
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'done'
      Collecting cffi
        Using cached cffi-1.15.1-cp311-cp311-macosx_10_9_x86_64.whl (179 kB)
      Collecting pycparser
        Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
      Installing collected packages: wheel, setuptools, pycparser, zope.interface, cffi, persistent
        DEPRECATION: zope.interface is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Di
        Running setup.py install for zope.interface: started
        Running setup.py install for zope.interface: still running...
        Running setup.py install for zope.interface: finished with status 'done'
        DEPRECATION: persistent is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Di
        Running setup.py install for persistent: started
        Running setup.py install for persistent: finished with status 'error'
        error: subprocess-exited-with-error
      
        × Running setup.py install for persistent did not run successfully.
        │ exit code: 1
        ╰─> [36 lines of output]
            /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
              warnings.warn(
            WARNING: The wheel package is not available.
            Traceback (most recent call last):
              File "<string>", line 2, in <module>
              File "<pip-setuptools-caller>", line 34, in <module>
              File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-m131xxtj/persistent_b1aa81d899754ef5b9950720f12c1317/setup.py", line 88, in <module>
                setup(name='persistent',
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/__init__.py", line 86, in setup
                _install_setup_requires(attrs)
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/__init__.py", line 80, in _install_setup_requires
                dist.fetch_build_eggs(dist.setup_requires)
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/dist.py", line 875, in fetch_build_eggs
                resolved_dists = pkg_resources.working_set.resolve(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pkg_resources/__init__.py", line 789, in resolve
                dist = best[req.key] = env.best_match(
                                       ^^^^^^^^^^^^^^^
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1075, in best_match
                return self.obtain(req, installer)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1087, in obtain
                return installer(requirement)
                       ^^^^^^^^^^^^^^^^^^^^^^
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/dist.py", line 945, in fetch_build_egg
                return fetch_build_egg(self, req)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/installer.py", line 87, in fetch_build_egg
                wheel.install_as_egg(dist_location)
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/wheel.py", line 110, in install_as_egg
                self._install_as_egg(destination_eggdir, zf)
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/wheel.py", line 118, in _install_as_egg
                self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
              File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/wheel.py", line 162, in _convert_metadata
                os.rename(dist_info, egg_info)
            OSError: [Errno 66] Directory not empty: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-m131xxtj/persistent_b1aa81d899754ef5b9950720f12c1317/.eggs/cffi-1.15.1-py3.11-macosx-10.9-universal2.egg/cffi-1.15.1.dist-info' -> '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-m131xxtj/persistent_b1aa81d899754ef5b9950720f12c1317/.eggs/cffi-1.15.1-py3.11-macosx-10.9-universal2.egg/EGG-INFO'
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: legacy-install-failure
      
      × Encountered error while trying to install package.
      ╰─> persistent
      
      note: This is an issue with the package mentioned above, not pip.
      hint: See above for output from the failure.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
er
ror: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Error: Process completed with exit code 1.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[xavfernandez]

It looks very similar to #10739

[pfmoore]

As noted in #10739, setting the option using an environment variable is a workaround.

For the general case, it's not obvious (to me, at least) that needing --use-pep517 in the main invocation implies that --use-pep517 is the right thing for creating the build environment - or vice versa. Longer term, I think we need a better design for how options get shared between the copy of pip that the user invoked, and the copies of pip used to set up the build environments. One that addresses questions like that.

[xavfernandez]

Longer term, I think we need a better design for how options get shared between the copy of pip that the user invoked, and the copies of pip used to set up the build environments.

#9081 sidesteps this issue :)

[ichard26]

It looks very similar to #10739

Yup! Also, I'd like to make progress on #9081 which would indeed involve formalising the interaction between the parent install flow and any build dep installs.