Why is data_files deprecated?

[hroncok]

I've noticed today that data_files is deprecated in https://setuptools.readthedocs.io/en/latest/references/keywords.html

It says:

Warning

data_files is deprecated. It does not work with wheels, so it should be avoided.

I've tried to git blame the info to know where did it originate, but apparently it was added with the entire keywords page.

When I read https://packaging.python.org/guides/distributing-packages-using-setuptools/#data-files I got no idea it is deprecated and I'd like to know:

• What is the reason? What does "does not work with wheels" even mean? Is it related to bdist_wheel makes absolute data_files relative to site-packages wheel#92? Because that only is about absolute paths. Using data_files with a relative path works like a charm. 😕
• What is the replacement?

Thanks.

[webknjaz]

I suppose that the reason is that it's a bad idea to manage files in random locations on the file system... But I'll let @jaraco elaborate on this.

Interestingly, this mentions a fix of that (probably) compatibility issue: https://setuptools.readthedocs.io/en/latest/history.html#v38-2-4.

@hroncok what is your use case, anyway? Can it be solved by using package_data? https://setuptools.readthedocs.io/en/latest/userguide/datafiles.html.

[hroncok]

Installing from wheels = unpacks the wheel and it works fine.

[FFY00]

What we do in arch is just install the files manually in these situations, I assume Fedora has similar issues with other ecosystems.

[FFY00]

Example: https://github.com/archlinux/svntogit-community/blob/9102cee0d987d5e723f3c3cf857b6e0f13878da3/trunk/PKGBUILD#L63
(in this case they could be installed by CMake but they aren't for some reason 🤷)

[hroncok]

We do that manually as well but people keep asking how to do it upstream.

[webknjaz]

Upstreams usually don't have RPM spec files.

My suggestion was that this should be done outside of the setuptools toolchain because while the source dist build executes code, the wheel install process doesn't. And I don't think it's fair to expect for static artifacts to mutate files outside of their location under site-packages. This mostly targets virtualenv-based workflows and RPM, OTOH, targets system-wide installations. So I think the former shouldn't try to get out of its isolation while it's okay for the latter.

[FFY00]

The reason it doesn't work with wheels and it did with eggs is because eggs could run code during installation, so you could operate directly on the filesystem.
Here's a recent discussion about the issue: https://discuss.python.org/t/should-there-be-a-new-standard-for-installing-arbitrary-data-files/7853

[fpatz]

Look, I have not asked you to comment on this on your weekend or at all, and for certain I didn't post here to get rude replies. I would have appreciated a discussion exchanging reasonable arguments, but you resorted to giggle on me, suggest misconceptions, naivety and whatnot on my side, while not presenting a single consistent argument except unfounded claims. I know how distros are opinionated about everything in the world and each other, but they still deal with all kinds of software and design decisions, and I very much doubt that their lifes become any better if Python deprecates (location-constrained) data files.

My position on this remains unchanged, I stand with all the points made in the previous comments for lack of any counter-arguments and I claim to have competently read the PEPs and decades of experience with Python packaging. You're totally free to ignore me, though.

[FFY00]

Sorry if my replies came out as rude, that was not my intention. What I was trying to say is that Python packaging is extremely complex and involved, not that you lack any competency.

Sorry for my giggle comment, it was poorly worded and did not provide enough context. I said that because I have been personally involved with dealing with the fall down of distros beeing opinionated.

That said, I did present a consistent argument. There is no reliable way to put data into sys.prefix. The .data directory in the PEP does not define any semantics apart from them being "install scheme keys", whatever you put in the data subdir is just not guaranteed to end up in sys.prefix. In practice, they will go to sys.prefix on a standard installation with the default install scheme, anything other than that might break your assumption.
The correct way to access data installed in .data/data is sysconfig.get_path('data'), not by using sys.prefix.

[fpatz]

Thank you for that reply, much appreciated on the Internets these days! I hope, I didn't spoil your Sunday evening by being so sensitive ;-)

And yes, you are right: the data directory goes to sysconfig.get_path("data"), which just happens to be sys.prefix in most cases. I was superficial and wrong about this. Given that, I still see no good reason to deprecate data files. The more precise differentiations in PEP491 -- e.g. $datarootdir possibly mapping to something like $prefix/share -- would be an improvement, though.

[FFY00]

Sorry for the delay.

Could you explain what is the point of installing to sysconfig.get_path('data') instead of purelib/platlib?

Cons:

• You may have conflicts, as in virtual environments for eg. it will generally point to the root of the environment
• People will often fall into the mistake of assuming that the data will be installed into the system, which is only true in some cases

The only pro I could think of is that it would allow to install to the system in certain cases, but that is not guaranteed, so you cannot rely on it, and it can quickly turn into a con, as I pointed out above. Is there anything other than this?

OTOH, installing to purelib/platlib is reliable and has none of these issues.

[fpatz]

You're right that package data is easier to reason about, and that is also what should be recommended to package authors in most cases. The use cases for data files are all about multiple more or less related Python packages that share some facility, where the facility cannot (efficiently) use package introspection to discover data files (generally, because it is somehow external to Python). PEP491 has exemplary use cases for this, like shipping man pages and other kinds of documentation, header files, etc. -- this, BTW, is exactly the same use case distros have with their packaging stack, and works that way for the same reason: distro packages install header files etc. into common places, not into some package directory.

Regarding conflicts: all Python packages can easily have conflicts today already, e.g. via entry-points or module files going directly to purelib. It is an illusion that we can neatly stuff everything into a single directory and get rid of all problems at once.

Regarding system vs virtualenv: given Python's most popular application domains, it is IMHO reasonable to assume that the primary use case for Python packaging are virtual environments. The Python community strongly recommends to not install to sytem, but use environments. Data files going into sys.prefix support the purpose of isolating environments.

[Thaodan]

The reason it doesn't work with wheels and it did with eggs is because eggs could run code during installation, so you could operate directly on the filesystem. Here's a recent discussion about the issue: https://discuss.python.org/t/should-there-be-a-new-standard-for-installing-arbitrary-data-files/7853

Will data_files work until there's an appropriate replacement? Considering that not even tools such as gettext work without installing data outside of the python packages directory.

[abravalheri]

Will data_files work ...

data_files will keep working as they are right now, there is no plan to remove this configuration parameter in the short-term.

Note however that there is no guarantee that the files you list will end up in a specific location of your system (e.g. /etc/something). This hasn't been supported in a long time and there is no plan to add support for that.

What will happen is that setuptools will add the files listed in data_files to the .whl archive it produces.

Later the installer tool (most likely pip) will extract those files on a location according to the most appropriate installation layout. Depending on the layout used by the installation tool, the files might end up in a different location. Different layouts options consider virtual environments, OS-specific variants, user's home directory, etc... The choice of the installation layout also depends on how the end use calls the installer tool, and which options they pass.

until there's an appropriate replacement?

The appropriate replacements are already defined:

• If you depend on a specific absolute path in the operating system, the replacement is native packaging solutions for that operating system (e.g. Debian => .deb, Fedora => .rpm, etc ...)
• If you are OK with the installation directory being relative to a Python virtual environment, you can still keep using data_files, or even better just store the data files inside regular Python package folders (which gives you integration with importlib.resources). Please see more info in https://setuptools.pypa.io/en/latest/userguide/datafiles.html.

[Thaodan]

Anderson Bravalheri ***@***.***> writes:

> Will data_files work ... `data_files` will keep working as they are right now, there is no plan to remove this configuration parameter in the short-term. Note however that there is no guarantee that the files you list will end up in a specific location of your system (e.g. `/etc/something`). This hasn't been supported in a long time and there is no plan to add support for that.

I was thinking that go to the path of `sys.prefix` such as those that go to `/usr/share` or `/usr/lib` (systemd, or default configuration).

What will happen is that setuptools will add the files listed in `data_files` to the `.whl` archive it produces. Later the installer tool (most likely `pip`) will extract those files on a location according to the most appropriate installation layout. Depending on the layout used by the installation tool, the files might end up in a different location. Different layouts options consider virtual environments, OS-specific variants, user's home directory, etc...

If they are installed into `sys.prefix` the behavior should be consistent, excluding exotic packaging systems such as Nix of course.

> until there's an appropriate replacement? The appropriate replacements are well known: - If you depend on a specific absolute path in the operating system, the replacement is native packaging solutions for that operating system (e.g. Debian => `.deb`, Fedora => `.rpm`)\

Of course that's a solution but it would be easier to do it in setuptools to not have to do it in each package manually. Besides tools such as setuptools-gettext exist too, maybe we just need a setuptools extension for each type of data that has to be installed outside of the python site.

[abravalheri]

The thing is: setuptools build the distribution files, but it does not install them (pip does the installation), so setuptools has no control over where the exact location where they are going to be installed. The same is true for setuptools extensions.

That is why, if you need to install something in a very specific location like /etc/something or /usr/something and you don't have the flexibility of placing these files inside a virtual environment), a setuptools/pip-centric solution is most likely not able to help you.