Improve README regarding how to sync "dev" requirements

[chrihartl]

I have a slight problem understanding a detail in the README file. In the section "Workflow for layered requirements" it basically says that I should do:

pip-compile
pip-compile dev-requirements.in
pip-sync # for prod
pip-sync requirements.txt dev-requirements.txt # for dev

My question is: Do I really need to provide both, dev-requirements.txt AND requirements.txt to pip-sync?

In my case, the dev-requirements.in already contains a reference to the base requirements (-c requirements.txt).
Therefore I would have thought that it is sufficient to do:

pip-sync dev-requirements.txt

Maybe this question could be addressed in the README?
I might just have overlooked the answer or misunderstood something, of course.

[aberres]

The important word here is "layered". In your case dev is an additional layer on top. It is not self contained.

-c ensures that the dev requirements use the same versions for packages which are referenced from both requirements files.

If you want dev-requirements to be self contained you could try -r requirements.txt.

[chrihartl]

Thanks, this makes it a bit clearer. I was not aware of the pip flag -c. I also read #398. But then I still wonder why in my example it seems to be sufficient to do

pip-sync dev-requirements.txt

or, really, why

pip-compile  # setup.py -> requirements.txt
pip-compile dev-requirements.in  # contains -c requirements.txt -> dev-requirements.txt

produces a list of dependencies that seem to be self-consistent: In dev-requirements.txt, every dependency constrained from requirements.txt is listed as "via: -c requirements.txt".

Following your explanation I would have expected instead that only those requirements from requirements.txt that are also listed in dev-requirements.in would be included in dev-requirements.txt (and constrained to the versions in requirements.txt). What did I miss?

(Actually I like the behavior that I get, but I would like to be sure about how it works.)

[aberres]

First I have to say, that I am just a user.

Following your explanation I would have expected instead that only those requirements from requirements.txt that are also listed in dev-requirements.in would be included in dev-requirements.txt

This is the behavior I am seeing in my setup. But not only for directly linked dependencies, but as exected also for transitive dependencies. For example

pyyaml==6.0
    # via
    #   -c requirements.txt
    #   pre-commit

pyyaml is a dependency of my packackes and as well comes in via pre-commit. Others are only found in requirements.txt.

With -r instead I see your described - and expected - behavior.

[chrihartl]

I think I understand now how it works. I believe that got confused because of my special setup. My dev-requirements.in looked similar to this:

-c requirements.txt
-e file:.#egg=mypackage

That is, when developing my package, I required the package itself (that corresponds to the Python project, for which I was about to pin dependency versions) to be installed in the virtual environment, in editable manner (because I was using console scripts that become available through installation).

But the transitive transitions of this very package simply correspond to the requirements.txt derived from setup.py using pip-compile. Therefore it is not suprising that pip-compile dev-requirements.txt is already self-contained in my special case -- despite using -c requirements.txt and not -r requirements.txt.

Thank you @aberres for helping me clean this up in my head.