Use pkg-config to find dependencies

[SantiagoCastro]

BPO	28207
Nosy	@tiran, @ned-deily, @zware, @koobs, @yan12125

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-09-19.19:34:50.693>
labels = ['extension-modules', 'type-feature', '3.7', 'build']
title = 'Use pkg-config to find dependencies'
updated_at = <Date 2019-12-10.08:13:29.738>
user = 'https://bugs.python.org/SantiagoCastro'

bugs.python.org fields:

activity = <Date 2019-12-10.08:13:29.738>
actor = 'xdegaye'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build', 'Extension Modules']
creation = <Date 2016-09-19.19:34:50.693>
creator = 'Santiago Castro'
dependencies = []
files = []
hgrepos = []
issue_num = 28207
keywords = []
message_count = 15.0
messages = ['276983', '277001', '277063', '277314', '277333', '277347', '277400', '277403', '277405', '277413', '277424', '277438', '277488', '277490', '277497']
nosy_count = 6.0
nosy_names = ['christian.heimes', 'ned.deily', 'zach.ware', 'koobs', 'yan12125', 'Santiago Castro']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28207'
versions = ['Python 3.7']

[SantiagoCastro]

When installing Python (for example, version 3.5.2), if SQLite library and headers are not installed in a default location (like $HOME/.local/include instead of /usr/include), it should take it from there, and not fail to find it. This behavior does work with bz2 library and OpenSSL, but not with SQLite.

[yan12125]

Version 3.5.2 works for me:

sqlite: found /home/yen/usr/include/sqlite3.h
/home/yen/usr/include/sqlite3.h: version 3.14.2

How did you compile CPython? Could you paste commands you use?

[SantiagoCastro]

I tried with pyenv (https://github.com/yyuu/pyenv): pyenv install 3.5.2. Maybe the error is from their side, but basically it downloads Python and compiles it: https://github.com/yyuu/pyenv/blob/master/plugins/python-build/install.sh#L24

[yan12125]

Could you try this:

CPPFLAGS=-I/home/<username>/local/include/ LDFLAGS=-L/home/<username>/local/lib bash -x /usr/bin/pyenv install 3.5.2

[SantiagoCastro]

Okay, that did work. But shouldn't it call pkg-config, so I don't need to set the flags manually?

I mean, I'm running this in my user's home, in a computer which I don't have root access, and I used Linuxbrew (https://github.com/Linuxbrew/brew) for this, and installed libbz2 headers, openssl headers, pkg-config, sqlite3 headers and all worked but finding the sqlite3 headers.

Just fyi, when I run pkg-config --cflags sqlite3 this is what I get:

-I/home/sacastro/.linuxbrew/Cellar/sqlite/3.14.1/include

which is ok.

[yan12125]

Hmmm, currently only _ctypes uses pkg-config to detect libffi's header path, as it's not easy to determine without pkg-config. Is there a magic that enables openssl and bz2 outside standard paths :)

Anyway, using pkg-config is not a bad idea. I'd like to hear from some core developers. Zach, is it a good idea to introduce pkg-config for dependencies?

[SantiagoCastro]

Okay, I checked out again and bz2 and openssl were in standard paths in fact, my bad. But I think python should also take into account pkg-config. I left a Dockerfile with how I think it should work with pyenv and Linuxbrew: https://github.com/bryant1410/docker-pyenv-linuxbrew

[zware]

I don't have any philosophical opposition to using pkg-config, but it would be nice to avoid making configure/Makefile/setup.py any more complex than they already are. If you can somehow simplify those files (without breaking anything) by using pkg-config, I'd be all for it :)

Resetting the version to 3.7, this feels like a pretty big change. If it turns out well, we can consider backporting it later.

[yan12125]

Thanks. I'll give it a try.

[tiran]

Simple implementation idea:

• add a pkg_config option to Extension()
• run subprocess.call(["pkg-config", self.pkg_config, "--exists"])
• if return value is 0, extend self.extra_compile_args with ["pkg-config", self.pkg_config, "--cflags"] and self.extra_link_args with ["pkg-config", self.pkg_config, "--libs"]
• otherwise print a warning

We could split and parse the output to handle -I, -L and -l in a more elegant way.

[zware]

Note the bootstrap issue with that idea though; you'll need to make sure _posixsubprocess is built before importing subprocess.

[ned-deily]

Any solution using pkg-config would need to take into account that pkg-config may not be available (by default) on platforms we support; for example, AFAIK, Apple does not ship pkg-config in the base OS or any of its developer tools. And the solution would need to be careful to not break the ability to use the subset of Distutils the top-level setup.py needs to bootstrap build the standard library. Also keep in mind that a lot of this proposed functionality is (or should be) available today by using Modules/Setup.local.

[yan12125]

Yes pkg-config is not ubiquitous. My idea is using it as a fallback, so that non-standard paths can be picked up easily.

Note the bootstrap issue with that idea though; you'll need to make sure _posixsubprocess is built before importing subprocess.

distutils.spawn.spawn or os.system can be used. setup.py uses the latter.

[koobs]

This (adding support for pkg-config) should be done in tandem with adding --with-foo-{include,library} arguments for each *external* dependency, which can be used for: libffi, readline, libintl, openssl, sqlite, db*, among others.

Values obtained from pkg-config should then use the same variables, but only if they are/have not been specified at the command line (./configure).

Doing this and a few other best-practice, standard autoconf things will enable us to drastically simplify and remove hacks in the Python configure.ac, Makefile and setup.py's long term.

[yan12125]

Thanks Kubilay I got it.

Note to myself: gdb uses lots --with-foo-{include,lib}.

[erlend-aasland]

Superseded by #89736 and #90005