Heisenbug when installing with --constraint #10824

pombredanne · 2022-01-24T22:15:05Z

Description

OS: macOS v11.6
pip: 21.2.3
Python 3.6.

ScanCode Toolkit is using a --constraint installation that fails sometimes. For instance on macOS with Python 3.6

wget https://github.com/nexB/scancode-toolkit/releases/download/v30.1.0/scancode-toolkit-30.1.0_py36-macos.tar.xz
tar -xf scancode-toolkit-30.1.0_py36-macos.tar.xz
cd scancode-toolkit-30.1.0
./configure --> this runs:
pip install --no-index --find-links `pwd`/thirdparty --upgrade --no-build-isolation --editable . --constraint requirements.txt
which fails sometimes with the cryptic message:

Processing ./thirdparty/zipp-3.4.1-py3-none-any.whl
ERROR: Cannot install scancode-toolkit==30.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
scancode-toolkit 30.1.0 depends on intbitset<3.0 and >=2.3.0
The user requested (constraint) intbitset==2.4.1

To fix this you could try to:

loosen the range of package versions you've specified

remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

This works fine 99% on all other machines and across OSes.
intbitset==2.4.1 is available as a pre-built wheel under thirdparty/ alright

I am at lost to find a way to diagnose the issue. These kind of issues started randomly with the new resolver.

The pip version used is the one bundled in the included https://github.com/pypa/get-virtualenv/raw/20.7.2/public/virtualenv.pyz
that I include in my etc/thirdparty for easy bootstrapping

Expected behavior

I would like to be able to trace the installation issue.

pip version

21.2.3

Python version

3.6

OS

macOS v11.6

How to Reproduce

ScanCode Toolkit is using a --constraint installation that fails sometimes. For instance on macOS with Python 3.6

wget https://github.com/nexB/scancode-toolkit/releases/download/v30.1.0/scancode-toolkit-30.1.0_py36-macos.tar.xz
tar -xf scancode-toolkit-30.1.0_py36-macos.tar.xz
cd scancode-toolkit-30.1.0
./configure

Output

See above for details

Code of Conduct

I agree to follow the PSF Code of Conduct.

The text was updated successfully, but these errors were encountered:

pombredanne · 2022-01-24T22:19:54Z

There is another report I just received on this at aboutcode-org/scancode-toolkit#2808
@DennisClark @lucasgonze ping

In this other report the dependency resolution error happens elsewhere on another dep.
And this is another mac OS X Monterey v12.1 and otherPython 3.9.9 version

The conflict is caused by:
scancode-toolkit 30.1.0 depends on lxml<5.0.0 and >=4.6.3
The user requested (constraint) lxml==4.6.3

pradyunsg · 2022-01-25T00:01:48Z

This might be an issue around wheel compatibility, that's manifesting as a dependency resolution issue?

lucasgonze · 2022-01-25T00:58:45Z

At some point in the process I was getting the same conflict but on a different dependency.

pombredanne · 2022-01-25T13:17:17Z

@pradyunsg

This might be an issue around wheel compatibility, that's manifesting as a dependency resolution issue?

It could well be, but how can I investigate and solve the issue?
The error message is not helping me doing a diagnostic.

I kinda grok a little the pip internals and I understand it is really hard to provide a proper actionable feedback. But if an experienced person cannot figure things out, then we have a problem to solve because that's IMHO an important breakage in the pip UX and a regression from before the introduction of the resolver in pip.

If there is any wheel incompatibility on any wheel, we should return a message saying so such as :

There is no available lxml==4.6.3 that can be installed from the list of available lxml==4.6.3 
[<here print a list of wheels or sdist with their index or link origin>] 
from these index and links [<here print a list of indexes and links>] 
that is compatible with this environment constraints of 
[here print a list of constraints as markers/extra/]  
[and available as a pre-built binary wheel | buildable from sources (if there are binary/no-binary)]

So, what would you suggest I do to diagnose and resolve this issue here?
Can I disable entirely dependency resolution and just force feed pip with a list of package requirements that I know work together?

And how can I help fix this pip UX otherwise?

pombredanne · 2022-01-25T13:18:54Z

@lucasgonze

At some point in the process I was getting the same conflict but on a different dependency.

yeah, this makes this a real heisenbug as the errors are not always the same.

notatallshaw · 2022-01-25T13:31:29Z

@lucasgonze

At some point in the process I was getting the same conflict but on a different dependency.

yeah, this makes this a real heisenbug as the errors are not always the same.

I wonder if that's related to this reproducible example: #10391 (comment)

In that case the user is able to get different conflict error messages to display on different runs, I have reproduced their example locally.

pombredanne · 2022-01-25T17:03:31Z

I can confirm that #10391 (comment) looks very similar.

pombredanne · 2022-01-25T21:50:01Z

@Hritik14 I recall you have a mac ... may be you could test this? This would be super gentle of you! ❤️

Hritik14 · 2022-01-25T22:46:24Z

@pombredanne Sure.
ProductName: macOS
ProductVersion: 12.1
BuildVersion: 21C52
Chip: Apple M1
Python: Python 3.9.10 (I couldn't get 3.6 to compile on Mac M1)

; wget https://github.com/nexB/scancode-toolkit/releases/download/v30.1.0/scancode-toolkit-30.1.0_py36-macos.tar.xz
; tar -xf scancode-toolkit-30.1.0_py36-macos.tar.xz
; cd scancode-toolkit-30.1.0
; PYTHON_EXECUTABLE=python3 ./configure

Although, I get a different package: MarkupSafe

Processing ./thirdparty/attrs-21.2.0-py2.py3-none-any.whl
Processing ./thirdparty/boolean.py-3.8-py2.py3-none-any.whl
ERROR: Cannot install scancode-toolkit==30.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
scancode-toolkit 30.1.0 depends on MarkupSafe>=1.0
The user requested (constraint) markupsafe==2.0.1

To fix this you could try to:

loosen the range of package versions you've specified

remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Hritik14 · 2022-01-25T23:01:44Z

To my surprise, I could manually do

 ; pip install thirdparty/MarkupSafe-2.0.1-cp36-cp36m-macosx_10_9_x86_64.whl

and the error now turns to

Processing ./thirdparty/Pygments-2.9.0-py3-none-any.whl
Processing ./thirdparty/jsonstreams-0.6.0-py2.py3-none-any.whl
Processing ./thirdparty/typecode-21.6.1-py3-none-any.whl
ERROR: Cannot install scancode-toolkit==30.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
scancode-toolkit 30.1.0 depends on pyahocorasick<1.5 and >=1.4.0
The user requested (constraint) pyahocorasick==1.4.2

To fix this you could try to:

loosen the range of package versions you've specified

remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Then this followed

 ; pip install pyahocorasick==1.4.2
 ; ./configure

Which fails again with

The conflict is caused by:
scancode-toolkit 30.1.0 depends on pyahocorasick<1.5 and >=1.4.0
The user requested (constraint) pyahocorasick==1.4.2

After a few iterations (of different packages and installing them manually). I finally arrived to intbitset==2.4.1 which fails to install with the following clang error.

pip install intbitset==2.4.1

Collecting intbitset==2.4.1
  Using cached intbitset-2.4.1.tar.gz (152 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six in ./venv/lib/python3.10/site-packages (from intbitset==2.4.1) (1.16.0)
Building wheels for collected packages: intbitset
  Building wheel for intbitset (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/neo/Contrib/test-scancode/scancode-toolkit-30.1.0/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-install-2qcvgjva/intbitset_f7af9bbeb5ca43deb2833c9d5beb39bf/setup.py'"'"'; __file__='"'"'/private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-install-2qcvgjva/intbitset_f7af9bbeb5ca43deb2833c9d5beb39bf/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-wheel-9iqqsi8s
       cwd: /private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-install-2qcvgjva/intbitset_f7af9bbeb5ca43deb2833c9d5beb39bf/
  Complete output (22 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-12-arm64-3.10
  copying intbitset/intbitset_helper.py -> build/lib.macosx-12-arm64-3.10
  copying intbitset/intbitset_version.py -> build/lib.macosx-12-arm64-3.10
  running egg_info
  warning: no files found matching '*.css' under directory 'docs/_themes'
  warning: no files found matching '*.css_t' under directory 'docs/_themes'
  warning: no files found matching '*.conf' under directory 'docs/_themes'
  warning: no files found matching '*.html' under directory 'docs/_themes'
  warning: no files found matching 'COPYING' under directory 'docs/_themes'
  warning: no files found matching 'README' under directory 'docs/_themes'
  warning: no files found matching '*.html' under directory 'docs/_templates'
  writing manifest file 'intbitset/intbitset.egg-info/SOURCES.txt'
  running build_ext
  creating build/temp.macosx-12-arm64-3.10
  creating build/temp.macosx-12-arm64-3.10/intbitset
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/neo/Contrib/test-scancode/scancode-toolkit-30.1.0/venv/include -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c intbitset/intbitset.c -o build/temp.macosx-12-arm64-3.10/intbitset/intbitset.o -O3 -march=core2 -mtune=native
  clang: error: the clang compiler does not support '-march=core2'
  error: command '/usr/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for intbitset
  Running setup.py clean for intbitset
Failed to build intbitset
Installing collected packages: intbitset
    Running setup.py install for intbitset ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/neo/Contrib/test-scancode/scancode-toolkit-30.1.0/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-install-2qcvgjva/intbitset_f7af9bbeb5ca43deb2833c9d5beb39bf/setup.py'"'"'; __file__='"'"'/private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-install-2qcvgjva/intbitset_f7af9bbeb5ca43deb2833c9d5beb39bf/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-record-x6ezrxj5/install-record.txt --single-version-externally-managed --compile --install-headers /Users/neo/Contrib/test-scancode/scancode-toolkit-30.1.0/venv/include/site/python3.10/intbitset
         cwd: /private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-install-2qcvgjva/intbitset_f7af9bbeb5ca43deb2833c9d5beb39bf/
    Complete output (24 lines):
    running install
    /Users/neo/Contrib/test-scancode/scancode-toolkit-30.1.0/venv/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    creating build
    creating build/lib.macosx-12-arm64-3.10
    copying intbitset/intbitset_helper.py -> build/lib.macosx-12-arm64-3.10
    copying intbitset/intbitset_version.py -> build/lib.macosx-12-arm64-3.10
    running egg_info
    warning: no files found matching '*.css' under directory 'docs/_themes'
    warning: no files found matching '*.css_t' under directory 'docs/_themes'
    warning: no files found matching '*.conf' under directory 'docs/_themes'
    warning: no files found matching '*.html' under directory 'docs/_themes'
    warning: no files found matching 'COPYING' under directory 'docs/_themes'
    warning: no files found matching 'README' under directory 'docs/_themes'
    warning: no files found matching '*.html' under directory 'docs/_templates'
    writing manifest file 'intbitset/intbitset.egg-info/SOURCES.txt'
    running build_ext
    creating build/temp.macosx-12-arm64-3.10
    creating build/temp.macosx-12-arm64-3.10/intbitset
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/neo/Contrib/test-scancode/scancode-toolkit-30.1.0/venv/include -I/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c intbitset/intbitset.c -o build/temp.macosx-12-arm64-3.10/intbitset/intbitset.o -O3 -march=core2 -mtune=native
    clang: error: the clang compiler does not support '-march=core2'
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/neo/Contrib/test-scancode/scancode-toolkit-30.1.0/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-install-2qcvgjva/intbitset_f7af9bbeb5ca43deb2833c9d5beb39bf/setup.py'"'"'; __file__='"'"'/private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-install-2qcvgjva/intbitset_f7af9bbeb5ca43deb2833c9d5beb39bf/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/75/t8rmj49d3pl1y_rk3vbjzk_r0000gn/T/pip-record-x6ezrxj5/install-record.txt --single-version-externally-managed --compile --install-headers /Users/neo/Contrib/test-scancode/scancode-toolkit-30.1.0/venv/include/site/python3.10/intbitset Check the logs for full command output.

pombredanne · 2022-01-26T06:36:38Z

@Hritik14 Thank you ++ for helping there!
Yes the repeated seemingly random errors is what is typical.

Could you try this too?

pip uninstall pip intbitset
pip install thirdparty/intbitset-2.4.1-cp36-cp36m-macosx_10_9_x86_64.macosx_10_10_x86_64.whl

because there is a prebuilt wheel version bundled.

pombredanne · 2022-01-26T06:37:38Z

With Python 3.9, use instead https://github.com/nexB/scancode-toolkit/releases/download/v30.1.0/scancode-toolkit-30.1.0_py39-macos.tar.xz that contain the proper macOS wheels

Hritik14 · 2022-01-26T15:51:01Z

@pombredanne
Anytime :). Here's what I got.

; python --version
Python 3.9.10
; ./configure
...
The conflict is caused by:
    scancode-toolkit 30.1.0 depends on bitarray<3.0.0 and >=0.8.1
    The user requested (constraint) bitarray==2.1.0
...
; . bin/activate
(scancode-toolkit-30.1.0) ; pip install thirdparty/bitarray-2.1.0-cp39-cp39-macosx_10_14_x86_64.whl
ERROR: bitarray-2.1.0-cp39-cp39-macosx_10_14_x86_64.whl is not a supported wheel on this platform.
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/neo/Contrib/test-scancode/scancode-toolkit-30.1.0/bin/python -m pip install --upgrade pip' command.
scancode-toolkit-30.1.0) ; pip install bitarray==2.1.0  # installed

Same iteration withmarkupsafe==2.0.1. Finally landing on intbitset==2.4.1 where neither the given wheel in thirdparty/ works nor the one from mypi.

pombredanne · 2022-01-27T07:49:33Z

@Hritik14 Thank you ++

pombredanne · 2022-01-27T07:50:17Z

It would be helpful to have run logs as mentioned in #10391 (comment)

Can someone please run with PIP_RESOLVER_DEBUG=1 in the environment, and post the output of a couple of runs as GitHub Gists?

mayeut · 2022-01-30T10:43:18Z

It seems you're trying to install x86_64 wheels on a mac running arm64.
@Hritik14, I guess that running pip debug will show universal2 and arm64 tags only for platform specific tags. Can you confirm ?
If the wheels provided in the thirdparty folder are meant to be installed on any mac, they should be built as universal2 wheels nowadays. If you want to achieve maximum compatibility (depending on user's pip version), they should even have multiple tags, probably, for bitarray, something like macosx_10_14_x86_64.macosx_11_0_arm64.macosx_11_0_universal2

jtran · 2022-04-20T21:22:20Z

I ran into this same problem. Here are the things that were the same.

error message, where the reported conflicting constraints didn't actually conflict:

ERROR: Cannot install Nuitka~=0.7.7 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested Nuitka~=0.7.7
    The user requested (constraint) nuitka==0.7.7

running it again worked
used a constraint file

Here are things that were different.

Python 3.9.12
pip 22.0.4
Debian bullseye docker container
the pip package reported as causing the problem

the command invoked had slightly different arguments

pip install --no-cache-dir -r requirements.txt -c constraints.txt

pombredanne · 2022-04-20T21:32:39Z

@jtran Thanks. May be the use of constraints (which are somewhat new) is part of the problem?

pombredanne added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jan 24, 2022

pombredanne mentioned this issue Jan 24, 2022

Mac install problem: conflicting dependencies aboutcode-org/scancode-toolkit#2808

Closed

pombredanne mentioned this issue Jan 25, 2022

Conflicting dependencies culprit is not displayed #10391

Closed

1 task

pradyunsg added state: needs eyes Needs a maintainer/triager to take a closer look and removed S: needs triage Issues/PRs that need to be triaged labels Jan 25, 2022

pombredanne mentioned this issue Feb 2, 2022

Cannot install extractcode[full]==21.7.23 because these package versions have conflicting dependencies aboutcode-org/scancode-toolkit#2809

Closed

notatallshaw mentioned this issue Feb 3, 2022

ResolutionImpossible error message does not distinguish between extras #10873

Closed

1 task

pombredanne mentioned this issue Feb 17, 2022

Cannot install / configure ScanCode 30.1.0 anymore to due dependency version conflict aboutcode-org/scancode-toolkit#2869

Closed

notatallshaw mentioned this issue Oct 11, 2024

uv dendency resolution upper-bound problem leading to a suboptimal resolution astral-sh/uv#8128

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Heisenbug when installing with --constraint #10824

Heisenbug when installing with --constraint #10824

pombredanne commented Jan 24, 2022

pombredanne commented Jan 24, 2022

pradyunsg commented Jan 25, 2022

lucasgonze commented Jan 25, 2022

pombredanne commented Jan 25, 2022

pombredanne commented Jan 25, 2022

notatallshaw commented Jan 25, 2022

pombredanne commented Jan 25, 2022

pombredanne commented Jan 25, 2022

Hritik14 commented Jan 25, 2022

Hritik14 commented Jan 25, 2022

pombredanne commented Jan 26, 2022

pombredanne commented Jan 26, 2022 •

edited

Loading

Hritik14 commented Jan 26, 2022 •

edited

Loading

pombredanne commented Jan 27, 2022

pombredanne commented Jan 27, 2022

mayeut commented Jan 30, 2022 •

edited

Loading

jtran commented Apr 20, 2022

pombredanne commented Apr 20, 2022

Heisenbug when installing with --constraint #10824

Heisenbug when installing with --constraint #10824

Comments

pombredanne commented Jan 24, 2022

Description

Expected behavior

pip version

Python version

OS

How to Reproduce

Output

Code of Conduct

pombredanne commented Jan 24, 2022

pradyunsg commented Jan 25, 2022

lucasgonze commented Jan 25, 2022

pombredanne commented Jan 25, 2022

pombredanne commented Jan 25, 2022

notatallshaw commented Jan 25, 2022

pombredanne commented Jan 25, 2022

pombredanne commented Jan 25, 2022

Hritik14 commented Jan 25, 2022

Hritik14 commented Jan 25, 2022

pombredanne commented Jan 26, 2022

pombredanne commented Jan 26, 2022 • edited Loading

Hritik14 commented Jan 26, 2022 • edited Loading

pombredanne commented Jan 27, 2022

pombredanne commented Jan 27, 2022

mayeut commented Jan 30, 2022 • edited Loading

jtran commented Apr 20, 2022

pombredanne commented Apr 20, 2022

pombredanne commented Jan 26, 2022 •

edited

Loading

Hritik14 commented Jan 26, 2022 •

edited

Loading

mayeut commented Jan 30, 2022 •

edited

Loading