modules: python: better handling of the Python paths for Debian

[FFY00]

Hardcoding the name is fragile, and enabling it based on the existence of
/etc/debian_version (as the is_debianlike helper does) will result in
incorrect paths if the Python binary is not provided by Debian.

Using the deb_system distutils scheme instead makes sure we use the
install path from the current interpreter, which Debian could change
between releases, and gives us the correct value on Python installations
that are not provided by Debian (eg. deadsnakes, Github Action Python,
etc.)

Do notice, though, that there is still no guarantee that these are the
correct paths, as they assume all schemes paths have the install prefix
as a base, see #9284.

Signed-off-by: Filipe Laíns lains@riseup.net

[codecov]

Codecov Report

Merging #9288 (83be616) into master (68eca11) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #9288      +/-   ##
==========================================
+ Coverage   66.85%   66.87%   +0.02%     
==========================================
  Files         386      388       +2     
  Lines       85062    85247     +185     
  Branches    17551    17515      -36     
==========================================
+ Hits        56869    57011     +142     
- Misses      23401    23494      +93     
+ Partials     4792     4742      -50     

Impacted Files	Coverage Δ
mesonbuild/modules/python.py	65.39% <ø> (-0.18%)	⬇️
mesonbuild/scripts/vcstagger.py	87.50% <0.00%> (-4.17%)	⬇️
compilers/cython.py	43.18% <0.00%> (-3.16%)	⬇️
interpreterbase/_unholder.py	68.42% <0.00%> (-3.01%)	⬇️
mesonbuild/interpreterbase/_unholder.py	68.42% <0.00%> (-3.01%)	⬇️
interpreterbase/interpreterbase.py	68.09% <0.00%> (-2.56%)	⬇️
mesonbuild/interpreterbase/interpreterbase.py	68.09% <0.00%> (-2.56%)	⬇️
compilers/rust.py	76.47% <0.00%> (-1.70%)	⬇️
mesonbuild/interpreterbase/decorators.py	89.10% <0.00%> (-1.56%)	⬇️
interpreterbase/decorators.py	89.10% <0.00%> (-1.01%)	⬇️
... and 64 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68eca11...83be616. Read the comment docs.

[xclaesse]

Gave this a try on my Windows machine too:

>>> import distutils.command.install
>>> distribution = distutils.dist.Distribution()
>>> install_cmd = distribution.get_command_obj('install')
>>> install_cmd.prefix = ''
>>> install_cmd.finalize_options()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python38\lib\distutils\command\install.py", line 344, in finalize_options
    self.convert_paths('lib', 'purelib', 'platlib',
  File "C:\Python38\lib\distutils\command\install.py", line 487, in convert_paths
    setattr(self, attr, convert_path(getattr(self, attr)))
  File "C:\Python38\lib\distutils\util.py", line 122, in convert_path
    raise ValueError("path '%s' cannot be absolute" % pathname)
ValueError: path '/Lib/site-packages' cannot be absolute

[xclaesse]

OOC, that's the plan with distutils upstream? They could remove the module completely, so we should guard the import behind try/except and fallback to sysconfig? Also, other than deb_system, are there any other case where sysconfig is not enough?

[FFY00]

Yes, 1sec. I will push my code to handle all that.

[xclaesse]

I'm also wondering if we should actually allow installing python modules outside of Meson's prefix. Thinking about Windows where they should go into C:\Python38\Lib\site-packages. But that's a bridge we could cross later.

[xclaesse]

I don't think we need a try inside a try, import distutils should always work on all platforms, right?

[FFY00]

Not after 3.12.

[FFY00]

Unless someone is naughty enough to create a Python package that provides the same module.

[xclaesse]

In that case we'll need to fix links_against_libpython() too, let's cross that bridge when/if we get there? OTOH, this is getting petty nitpicking I admit ;P

OOC, how confident are we that it's going to be removed in 3.12? It has been formally announced already? When that happens, what's the likely outcome for Debian, they'll patch sysconfig instead?

Sorry again for all my questions, I wasn't familiar with this topic, so I like to learned it a bit :)

[FFY00]

In that case we'll need to fix links_against_libpython() too, let's cross that bridge when/if we get there? OTOH, this is getting petty nitpicking I admit ;P

I'd prefer doing in a different PR as it is not actually an issue right now, while this is. We can fix the current handling of the Debian install paths and then explore how to fase out distutils entirely.

OOC, how confident are we that it's going to be removed in 3.12?

Very.

It has been formally announced already? When that happens, what's the likely outcome for Debian, they'll patch sysconfig instead?

Yes, my best guess is either PEP 668 or that, or both.
Also related: https://bugs.python.org/issue43976 (maybe skip down to the latest discussion)

Sorry again for all my questions, I wasn't familiar with this topic, so I like to learned it a bit :)

No worries, I have been dealing with these issues for a little while and still struggle to keep track of everything.

[FFY00]

@xclaesse can you have a look? I think this is essentially ready.

[FFY00]

I pushed a fix hopefully dealing with the failing tests due to old Python versions.

@eli-schwartz I will submit the improvements to the message in a different pull request, if that's alright with you, as we probably want to have a different discussion and review for that. Hopefully, we can merge this ASAP.

[FFY00]

Sorry for the delay, I have been busy. I removed the try .. except block for the Debian stuff from this PR, I'll see if I have time to open a new PR next week.

[xclaesse]

Looks good to me, thanks!

[eli-schwartz]

Thanks! Indeed this looks good to me too.