Cannot install / configure ScanCode 30.1.0 anymore to due dependency version conflict

[sschuberth]

Description

Building our Docker image which includes ScanCode 30.1.0 (built from source) started to fail today with

#14 7.357 * Configuring ScanCode for first use...
#14 10.352 created virtual environment CPython3.8.10.final.0-64 in 2801ms
#14 10.352   creator CPython3Posix(dest=/usr/local/scancode-toolkit-30.1.0, clear=False, no_vcs_ignore=True, global=False)
#14 10.352   seeder PipInvoke(download=False, pip=embed, setuptools=embed, wheel=embed)
#14 10.352     added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
#14 10.352   activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
#14 10.793 Looking in links: https://thirdparty.aboutcode.org/pypi
#14 10.794 Obtaining file:///usr/local/scancode-toolkit-30.1.0
#14 10.796     Preparing wheel metadata: started
#14 12.640     Preparing wheel metadata: finished with status 'done'
#14 13.101 ERROR: Cannot install scancode-toolkit==30.1.0 because these package versions have conflicting dependencies.
#14 13.101 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
#14 13.101 
#14 13.101 The conflict is caused by:
#14 13.101     scancode-toolkit 30.1.0 depends on MarkupSafe>=1.0
#14 13.101     The user requested (constraint) markupsafe==2.0.1
#14 13.101 
#14 13.101 To fix this you could try to:
#14 13.101 1. loosen the range of package versions you've specified
#14 13.101 2. remove package versions to allow pip attempt to solve the dependency conflict

How To Reproduce

RUN curl -ksSL https://github.com/nexB/scancode-toolkit/archive/v$SCANCODE_VERSION.tar.gz | \
        tar -zxC /usr/local && \
        # Trigger ScanCode configuration for Python 3 and reindex licenses initially.
        cd /usr/local/scancode-toolkit-$SCANCODE_VERSION && \
        PYTHON_EXE=/usr/bin/python3 /usr/local/scancode-toolkit-$SCANCODE_VERSION/scancode --reindex-licenses && \
        chmod -R o=u /usr/local/scancode-toolkit-$SCANCODE_VERSION && \
        ln -s /usr/local/scancode-toolkit-$SCANCODE_VERSION/scancode /usr/local/bin/scancode

System configuration

• What OS are you running on? Linux / Docker (adoptopenjdk:11-jdk-hotspot-focal)
• What version of scancode-toolkit was used to generate the scan file? 30.1.0
• What installation method was used to install/run scancode? source download

[sschuberth]

Interestingly, the conflicting dependencies seem to change every time I try to build:

#14 14.058 The conflict is caused by:
#14 14.058     scancode-toolkit 30.1.0 depends on pygmars>=0.7.0
#14 14.058     The user requested (constraint) pygmars==0.7.0

[AyanSinhaMahapatra]

Our CI at azure is also failing on some cases because of this btw, like https://dev.azure.com/nexB/scancode-toolkit/_build/results?buildId=5637&view=results. Re-running solves this (only on CI) sometimes.

[pombredanne]

@sschuberth there has been some PyPI server issue following some CI misconfiguration. This should be resolved now.
We are also updating the settings so this never happens again. So let's keep this open.

[pombredanne]

@sschuberth Part of it may also be due to this pypa/pip#10824 and this pypa/pip#10391 issues for pip. This is being researched!

[pombredanne]

@sschuberth this works fine for me:

$ export SCANCODE_VERSION=30.1.0pombreda@omputer2:~/tmp/pi/tmp/rela$ curl -ksSL https://github.com/nexB/scancode-toolkit/archive/v$SCANCODE_VERSION.tar.gz | \
>         tar -zx
$ cd scancode-toolkit-$SCANCODE_VERSION
$ ./scancode -h

There is no need to reindex licenses anymore when you use a proper release archive BUT I see you are NOT using a release archive here but a GitHub generated git tarball. I would NOT recommend this way of installing as this is never something that is tested.

Use instead a proper release archive for your Python version and OS such as https://github.com/nexB/scancode-toolkit/releases/download/v30.1.0/scancode-toolkit-30.1.0_py38-linux.tar.xz
See all the ones listed on https://github.com/nexB/scancode-toolkit/releases/tag/v30.1.0

@tafli ping since you referenced this issues too.

[sschuberth]

I see you are NOT using a release archive here but a GitHub generated git tarball.

Yes, that's something we did in the past as not all releases we were interested in had binaries attached.

I would NOT recommend this way of installing as this is never something that is tested.

Out of curiosity, how's installing from a GitHub generated source archive different from installing from sources as documented (and thus probably tested) here?

[pombredanne]

Out of curiosity, how's installing from a GitHub generated source archive different from installing from sources as documented (and thus probably tested) here?

Good point: it should not be vastly different but there are no test for it. And since we are moving to a git-tag-derived automated versioning, this may have unwanted side effects as it may not work exactly as expected, though we do have a .VERSIOn generated for git tarballs to possibly mitigate this as seen here https://github.com/nexB/scancode-toolkit/blob/7bc0782fdfda9da5dba0500446ff3e8d58623e99/.gitattributes#L13 and here https://github.com/nexB/scancode-toolkit/blob/7bc0782fdfda9da5dba0500446ff3e8d58623e99/.VERSION .

Furthermore, we will eventually soon have a fairly different layout between a checkout and built distribution, in particular for installation speedups so a developer checkout will be best used for development only.

[sschuberth]

Thanks for the explanations.

@sschuberth this works fine for me:

Ok, let's close this then.