pip ignores the --cert argument when fetching build dependencies

[guillaumerenault]

Environment

• pip version: 20.2.3
• Python version: 3.7.9
• OS: docker image (Ubuntu 18.04)

Description

SSLCertVerificationError when pip tries to install build dependencies.
When supplying --index-url URL and --cert PATH, pip uses both argument when fetching the requested packages but not when fetching build dependencies.

The supplied --cert argument is not forwarded when attempting to download the build dependencies whereas the --index-url/-i is.

Logs (sensitive information redacted) :

ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wbotwze8/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://internal-mirror.net/api/pypi/repository/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.

root@ed7da1517ca1:/# python -m pip install -U --index-url https://internal-mirror.net/api/pypi/repository/simple     --cert /etc/ssl/certs/ca-certificates.crt    black
Looking in indexes: https://internal-mirror.net/api/pypi/repository/simple
Collecting black
  Using cached https://internal-mirror.net/api/pypi/repository/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz (1.1 MB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wbotwze8/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://internal-mirror.net/api/pypi/repository/simple -- 'setuptools>=41.0' setuptools-scm wheel
       cwd: None
  Complete output (9 lines):
  Looking in indexes: https://internal-mirror.net/api/pypi/repository/simple
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  Could not fetch URL https://internal-mirror.net/api/pypi/repository/simple/setuptools/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='internal-mirror.net', port=443): Max retries exceeded with url: /api/pypi/repository/simple/setuptools/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))) - skipping
  ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
  ERROR: No matching distribution found for setuptools>=41.0
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wbotwze8/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://internal-mirror.net/api/pypi/repository/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.

When running the problematic command alone, the same error occurs as the --cert option is not supplied:

root@ed7da1517ca1:/# /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-tq9w5igb/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://internal-mirror.net/api/pypi/repository/simple -- 'setuptools>=41.0' setuptools-scm wheel
Looking in indexes: https://internal-mirror.net/api/pypi/repository/simple
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
Could not fetch URL https://internal-mirror.net/api/pypi/repository/simple/setuptools/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='internal-mirror.net', port=443): Max retries exceeded with url: /api/pypi/repository/simple/setuptools/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))) - skipping
ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
ERROR: No matching distribution found for setuptools>=41.0

Expected behavior

• Fetch the packages using the provided arguments --index-url and --cert
• Fetch the build dependencies (setuptools...) also using the formerly provided arguments --index-url and -cert
• Install build dependencies
• Install packages

How to Reproduce

1. Start a debian based docker image with Python (docker run python:3.7)
2. Install black from your internal https mirror (using the appropriate --index-url MIRROR_URL and --cert CA_BUNDLE_PATH) : python -m pip install -U --index-url https://internal-mirror.net/api/pypi/repository/simple --cert /etc/ssl/certs/ca-certificates.crt black
3. pip will sucessfully download black but fail due to a SSLError when installing build dependencies (setuptools) as the provided --cert argument doesn't seem to be taken into account when fetching build dependencies.

Output

root@ed7da1517ca1:/# python -m pip install -U --index-url https://internal-mirror.net/api/pypi/repository/simple     --cert /etc/ssl/certs/ca-certificates.crt    black
Looking in indexes: https://internal-mirror.net/api/pypi/repository/simple
Collecting black
  Using cached https://internal-mirror.net/api/pypi/repository/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz (1.1 MB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wbotwze8/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://internal-mirror.net/api/pypi/repository/simple -- 'setuptools>=41.0' setuptools-scm wheel
       cwd: None
  Complete output (9 lines):
  Looking in indexes: https://internal-mirror.net/api/pypi/repository/simple
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /api/pypi/repository/simple/setuptools/
  Could not fetch URL https://internal-mirror.net/api/pypi/repository/simple/setuptools/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='internal-mirror.net', port=443): Max retries exceeded with url: /api/pypi/repository/simple/setuptools/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))) - skipping
  ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
  ERROR: No matching distribution found for setuptools>=41.0
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wbotwze8/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://internal-mirror.net/api/pypi/repository/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.

[ichard26]

Merging into #5502.