Requirement version constraints are too strict

[dvzrv]

Hi! When trying to package this project for Arch Linux I will have to remove a lot of overly strict version restrictions in the requirement files.

E.g. requirements.txt is read in setup.py which leads to these requirements becoming hard runtime boundaries.

This is very problematic in a packaging context, as we can not guard against upgrades to these dependencies in a rolling release distribution (but that is what unit tests of the respective dependents are for and are run for in a packaging context) and we can also not pin them, as we do not provide python packages in a venv. In this context version pinning (==) and upper boundaries (< or <=) that do not reflect what this project is actually compatible with and could be tested against (e.g. in CI) are very problematic (this is btw. also problematic for users not using this project in a venv, as they will have conflicting versions of the given dependencies installed eventually).

Please make these version requirements less strict and ideally continuously test this project in CI against higher versions of all dependencies, so that breakage is easily mitigated while still being able to have very loose upper boundaries.

[mstarecek]

Some requirements version constraints might be relaxed, that's for sure. But that's not always possible. Some of our dependencies broke backward compatibility or introduce bugs themselves. Therefore there will always be upper bound restrictions.
The question is what is "loose enough".

[saper]

There are two conflicting goals:

• integrators (like myself) that use SPSDK as a library demand flexibility because they need to satisfy other dependencies
• end-users who just for example use the command line and need to be have a stable, reliable working system (and NXP certainly would like to provide that)

The solution is here: https://caremad.io/posts/2013/07/setup-vs-requirement/

In short, we should remove the setup.py code that loads requirements.txt and stuffs them into setup.py.

The hard part is to determine which requirements.txt rules origin from the real life known breakage - like this one

# There's an issue with 0.3.0 on win
# https://github.com/pyocd/cmsis-pack-manager/issues/176
cmsis-pack-manager<0.3.0

or just a precautionary thing ("this version has worked for us and has been tested")

It is not always clear - for example 29e7821 does not say what went broken?

This way the release managers could be even more strict - simply pip freeze all requirements.txt for every release.

Next steps:

• move current requirements.txt into setup.py install_requires
• relax setup.py dependency rules that are precautionary only
• (if release managers find it good) pip freeze > requirements.txt

Having said that, if bincopy 17.10 does not really work, it should be fixed (either in SPSDK if we use it incorrectly or upstream) and the current requirements.txt rule is correct and should be moved to setup.py if we go on with this proposal (until this is resolved).

@mwsk

[dvzrv]

1.7.0 introduced a lot of strict == requirements, which can not be met outside of a venv.
As a downstream I have to manually remove all of those, as I can otherwise not package this software reliably (or at all).

[dvzrv]

Is anyone still addressing this? Otherwise I might as well just close this issue and just remove all the problematic version pinning manually.

[PureTryOut]

I rather not see this closed but resolved properly. As a package for another Linux distribution (Alpine Linux) this is an issue I care for and I would like a good upstream resolution rather than removing the pinning manually downstream forever.

[BohdanMarek]

From version SPSDK1.9, the latest versions of our dependencies started being regularly tested and then subsequently updated in SPSDK. This will loose the upper boundaries and hopefully help you with integration of SPSDK to your projects.

[RaitoBezarius]

SPSDK version ranges are still too strict IMHO, on 2.1.0, I see:

┃        >   - platformdirs<4.2,>=3.9.1 not satisfied by version 4.2.0
┃        >   - typing-extensions<4.10 not satisfied by version 4.10.0

And it seems that relaxing makes the tests all pass.

On 2.1.1, same for cryptography.

[BohdanMarek]

Hi @RaitoBezarius ,
SPSDK project is regularly tested and updated with the latest versions of it's dependencies.

Regarding the packages you mentioned, please see the actual release dates:
platformdirs 4.2.0 -> 31/01/2024
typing-extensions 4.10.0 -> 25/02/2024
SPSDK 2.1.0 -> 02/02/2024

SPSDK adapted latest versions of its dependent packages just few days before release of version v2.1.0.

I actually believe that our dependences are loose enough. You can check the latest release(v2.2.0). Please let me know if it satisfied your request.

[pwl8]

Hi,
I have the same issue: upper restriction is to strict. I have a different python package with similar approach and it is quite difficult to handle them both on one environment.
If you look into requirements of some most popular python packages, it is quite unusual to see such strict upper pin to particular versions.
Maybe loosing to major versions in most cases would be a better idea?