Install with poetry install broken due to missing pip executable

[carmenbianca]

Details

• Read the Docs project URL: https://readthedocs.org/projects/reuse
• Build URL (if applicable): https://readthedocs.org/projects/reuse/builds/25578647/
• Read the Docs username (if applicable): https://readthedocs.org/profiles/carmenbianca/

Expected Result

Using the following config:

build:
  os: ubuntu-22.04
  tools:
    python: "3.9"
  jobs:
    post_create_environment:
      - python -m pip install poetry
    post_install:
      - VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH python -m poetry install --with
        docs

The build should successfully use Poetry, as documented here

Actual Result

The build fails on the poetry install step.

The build also fails when removing the python -m substrings from the above configuration.

This is the output of the failed step:

Installing dependencies from lock file

Package operations: 49 installs, 9 updates, 0 removals

  - Downgrading certifi (2024.8.30 -> 2024.6.2)
  [... shortened for your convenience]
  - Installing sphinxcontrib-apidoc (0.5.0)

Installing the current project: reuse (4.0.3)
Preparing build environment with build-system requirements poetry-core>=1.1.0

Command ['/tmp/tmp9lc98fl8/.venv/bin/python', '/home/docs/checkouts/readthedocs.org/user_builds/reuse/envs/latest/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-24.2-py3-none-any.whl/pip', 'install', '--disable-pip-version-check', '--ignore-installed', '--no-input', 'poetry-core>=1.1.0'] errored with the following return code 2

Output:
/tmp/tmp9lc98fl8/.venv/bin/python: can't open file '/home/docs/checkouts/readthedocs.org/user_builds/reuse/envs/latest/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-24.2-py3-none-any.whl/pip': [Errno 2] No such file or directory

This started occurring at some point after 2024-09-05 (latest successful build), with 2024-09-10 as the earliest build that fails. There were no relevant changes to Sphinx, the RTD config, Poetry, dependencies, etc etc etc between these two dates.

Specifically, what appears to happen is this: As part of running poetry install, the build-system dependencies from pyproject.yaml must be installed. Snippet:

[build-system]
requires = ["poetry-core>=1.1.0"]
build-backend = "poetry.core.masonry.api"

As far as I can divine, Poetry (or Python?) delegates the installation of this dependency to Pip, and Pip then fails to install poetry-core because pip does not exist at a given path. I assume that:

• Pip is either not installed in $READTHEDOCS_VIRTUALENV_PATH;
• or it somehow divines a completely incorrect path where pip is (not) located.

[carmenbianca]

For reasons that are far beyond me, running poetry lock to re-generate poetry.lock has fixed the issue. See this PR: fsfe/reuse-tool#1081

Some notes:

• The commit of the last successful build did NOT build anymore when I attempted to build that commit again. Compare:
  • https://readthedocs.org/projects/reuse/builds/25534005/ (last successful, 2024-09-05)
  • https://readthedocs.org/projects/reuse/builds/25767550/ (same commit, failed, 2024-09-27)
  • The commit in question
• I used Poetry 1.3.x to generate the poetry.lock files.

I don't know if this is a bug on RTD's part. I will leave the issue open even though it is resolved for me, because this seems rather unusual to me, and the error message appears completely unrelated.

[carmenbianca]

This is still a bug.

[humitos]

You will able to use "partial overrides" when #11710 gets merged, with something like the following:

build:
  os: ubuntu-22.04
  tools:
    python: "3.9"
  jobs:
    create_environment:
      - python -m pip install poetry
    install:
      - python -m poetry install --with docs

[humitos]

We have documented how to use poetry in our documentation: https://docs.readthedocs.com/platform/latest/build-customization.html#install-dependencies-with-poetry