Not able to upgrade version for transitive Dependency

[ayushi-singhal]

I hope this is correct forum for this issue.

Environment Versions

1. OS Type - macOS
2. Python version: $ python -V - 3.8
3. pip version: $ pip --version - 24.0
4. pip-tools version: $ pip-compile --version - 2.0.2

Steps to replicate

1. I've two files requirements.in listing all dependencies and versions, and another is requirements.txt which is created after compiling requirements.in and is used to install dependencies.
2. certifi version is 2021.10.8 and mentioned in requirements.txt file. I am trying to upgrade the transitive dependencies like sentry, requests, etc in requirements.in file and but the certifi version is not changing in requirements.txt and virtual env.
3. Similarly, tornado is dependent on flower. After upgrading flower version in requirements.in file, running pip-compile and pip install does not change version for tornado in requirements.txt file.

I am not sure what steps or process I am missing here. I have tried using force cmd, cleared cache, re-installed virtual env.
I appreciate the help, Thanks.

Expected result

Lib version should get changed in requirements.txt.

Actual result

Lib versions are still old ones.

[AndydeCleyre]

Thanks! Can you provide the file contents, so I can try to reproduce?

[ayushi-singhal]

I may not be able to share the entire content of the file due to confidentiality.

requirements.in file:
flower==1.1.0

requirements.txt file:
flower==1.1.0
# via -r packages/shared/requirements/requirements.in
tornado==6.1
# via flower

Now upgrading flower==2.0.1, but it does not change tornado version after pip-compile, but only flower version to 2.0.1 in txt file.

[ayushi-singhal]

I can either add tornado lib in .in file and pip-compile if it is not updated via flower.

[webknjaz]

Did you forget to use -P?

[ayushi-singhal]

What is that for? Should it be used in pip-compile commands?

[webknjaz]

Yep

[ayushi-singhal]

I am not sure what it does and how it is used. I can try.

[ayushi-singhal]

The pip-compile command does not have a -p flag. It did no change for me.

[webknjaz]

Not -p but -P: https://pip-tools.rtfd.io/en/stable/cli/pip-compile/ / https://pip-tools.rtfd.io/en/stable/#updating-requirements.

[ayushi-singhal]

Oh wow! I used pip-compile --output-file requirements.txt requirements.in --upgrade-package tornado==6.4 and it worked. To upgrade transitive dependency, I am not sure this is acceptable.
For example, certifi lib is mentioned in requirements.txt file only via sentry-sdk, mailchimp-transactional and requests.
requests lib is again a transitive dependency mentioned in requirements.txt only via other libs.

To avoid any miss in upgrading any required lib, I deleted requirements.txt file and generated a fresh one from scratch and it updated versions for a lot of libs, which may not even be required.

[ayushi-singhal]

Would you please guide me how to update a lib version using transitive dependency, or the way I have done it is also acceptable?

[ayushi-singhal]

requirements.in file:

sentry-sdk==1.5.8
mailchimp_transactional==1.0.47

requirements.txt file:

certifi==2021.10.8
# via
# mailchimp-transactional
# requests
# sentry-sdk
requests==2.27.1
# via
# googlemaps
# mailchimp
# mailchimp-transactional
sentry-sdk==1.5.8
# via -r packages/shared/requirements/requirements.in

[webknjaz]

Yes, use this CLI flag for any package in the tree.

[webknjaz]

If your project is on GitHub, you can also enable Dependant — it is able to update pairs of files with the name base name, but different extensions — .in and .txt.

[ayushi-singhal]

I am not much clear about this solution, could you please elaborate on this?

I think I would not want to enable Dependant in Github repo before proper analysis about what it does and how it will work.

For now, I think I can upgrade required libs only.

If I upgrade version of one lib and it does not give conflicting dependency error, am I good to proceed or I need to make sure txt file is freshly generated every time?

[webknjaz]

Yes, the output is always correct. As for dependabot, here's and example of what PRs it sends: #2043.

[ayushi-singhal]

Thanks @webknjaz so much for your inputs. :) Appreciated!

[ayushi-singhal]

I have one more question.
upon deleting requirements.txt file and generating a fresh one using pip-compile, changes a lot of library/ dependency version which do not reflect on pip-compile normally.

I did not change anything in requirements.in file at all.

Why is that?

[webknjaz]

Because it doesn't upgrade the deps that don't necessarily need it, unless you ask.

[ayushi-singhal]

Thanks.

[ayushi-singhal]

One more query that I am facing for a week now.
Let's say I checkout the repo code, have made no change in requirements.in and requirements.txt file but while doing pip-compile, requirements.txt file is entirely updated. Why so?
Ideally requirements.txt file should have no changes.

[AndydeCleyre]

If no upgrade flag was passed, it could be that some locked version got revoked on PyPI, so new versions were needed. Other than that, I'm not sure what happened without seeing the initial .in and .txt files.

[webknjaz]

Could be that you're running it under a different Python version / OS / arch, I suppose. Or it could be coming from the config file. We can only fortune-tell having the entire context, by logs and other info 🔮

[ayushi-singhal]

I was earlier using Python 3.7 version but now it is 3.8.
But other people are using same config as mine, still they are not seeing any change in txt file.

It is intermittent for me as well. For once, I did not see any txt file changes.

What other info can I share here for your reference?

[webknjaz]

What other info can I share here for your reference?

Only a full reproducer. With versions of pip-tools, requirements input and output file, versions of the OS/distro, python version, version of pip. Ideally, in a form of a (docker) container that clearly shows the problem.

4. pip-tools version: $ pip-compile --version - 2.0.2

I only now realized that you seem to be using this ancient version of pip-tools. We don't support old versions. Neither do we support an old pip. That version doesn't even have a backtracking resolver and the legacy one can yield less stable behavior.

[ayushi-singhal]

versions of the OS/distro: macOS Monterey
python version: 3.8
version of pip: 24.0
version of pip-tools: 2.0.2

Unfortunately, I won't be able to share requirements input and output file. I understand that the above info is insufficient, but I can try to upgrade the version of pip-tools.

[webknjaz]

Don't forget to upgrade pip.