[BUG]: Embedded python doesn't include full sys.path

[danielcjacobs]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.3

Problem description

UPDATE: The following error seems to be related to #3923, which updated the scoped_interpreter constructor to initialize the python interpreter in an isolated environment via PyConfig_InitIsolatedConfig. If I init my own PyConfig object via PyConfig_InitPythonConfig and pass it to the scoped_interpreter constructor, I don't have the error. Thus, the issue is probably just a wrong assumption in how I'm supposed to use scoped_interpreter.

On the latest pybind, I'm trying to use the embedded python in C++.

Specifically, I'm trying to import matplotlib, which was installed to ~/.local/python3.11/site-packages via python -m pip install matplotlib

However, I'm getting the following error when running pybind11::module::import("matplotlib");

terminate called after throwing an instance of 'pybind11::error_already_set'
  what():  ModuleNotFoundError: No module named 'matplotlib'

After some debugging, it appears to be due to a difference between pybind 2.9.2 and pybind 2.10.0, as the following snippet has different outputs between the 2 releases:

auto sys  = pybind11::module::import("sys");
auto path = sys.attr("path");
pybind11::print(pybind11::str(path));

On 2.9.2, the output is:

`['/usr/lib64/python311.zip', '/usr/lib64/python3.11', '/usr/lib64/python3.11/lib-dynload', '~/.local/lib/python3.11/site-packages', '/usr/local/lib64/python3.11/site-packages', '/usr/local/lib/python3.11/site-packages', '/usr/lib64/python3.11/site-packages', '/usr/lib/python3.11/site-packages']`

But on 2.10.0, the output is missing a couple paths (namely, the local and .local paths):

`['/usr/lib64/python311.zip', '/usr/lib64/python3.11', '/usr/lib64/python3.11/lib-dynload', '/usr/lib64/python3.11/site-packages', '/usr/lib/python3.11/site-packages']`

Environment:

• OS: Fedora 37
• Python: 3.11.1
• Pybind: 2.10.0 and later
• Matplotlib: 3.6.3
• Compiler: gcc 12.2.1

Reproducible example code

#include <pybind11/embed.h>
#include <pybind11/pybind11.h>

int main() {
	pybind11::scoped_interpreter guard{};

	auto sys  = pybind11::module::import("sys");
	auto path = sys.attr("path");
	pybind11::print(pybind11::str(path));

	// fails
	auto pyplot   = pybind11::module::import("matplotlib");
}

Is this a regression? Put the last known working version here if it is.

Possibly. Worked in 2.9.2

[danielcjacobs]

After more digging, it looks like the problem is that the default scoped_interpreter constructor initializes the interpreter in an isolated environment (as described here).

I'm not sure where exactly this change was first introduced (since it worked before), but I suppose my question is:

Is it better practice for me to initialize my own PyConfig and pass it to the scoped_interpreter constructor, or to find a way to make it work via the default constructor?

Also, it seems weird that this line sets up an isolated config, but then config.isolated is set to 0. Perhaps it'd be better to change it use PyConfig_InitPythonConfig instead of PyConfig_InitIsolatedConfig