Broken Conda macOS Python interpreter

[lohedges]

(This is a known issue and, at present, I'm not sure what we can do about it. I'm just documenting it here for posterity. References to issues can be found here and here.)

The current macOS Python 3.7 interpreter from the conda-forge channel does not work with Sire, which segmentation faults on import. Importing Sire works fine using the bundled sire_python interpreter, with the current macOS Python 3.7 and 3.7m interpreters from python.org, and with the simplest home-cooked Python interpreter that you can make, e.g.:

// test.cpp
#include <Python.h>

int main()
{
    wchar_t* argv;
    int status = Py_Main(0, &argv);
    return status;
}

Compile with:

g++ test.cpp -I $HOME/sire.app/include/python3.7m -L $HOME/sire.app/lib -lpython3.7m -o $HOME/sire.app/bin/test_python

Then launch the interpreter:

$HOME/sire.app/bin/test_python

Finally import Sire:

import Sire

No errors. Ridiculous!

I'll post a report on conda-forge to see if they can figure out what aspect of their Python 3.7 build is causing the issue for us.

[lohedges]

Okay, the problem is that, as @chryswoods mentions here, libpython is statically linked into the macOS Python interpreter, which means that any compiled Python extensions CANNOT link to libpython for fear of duplicating symbols. Previously this was leading to the "no current thread issue that @ppxasjsm was seeing.

To see this we can create our own Python interpreter with a static link of libpython, i.e. using the same test.cpp code above. (Here I build on Linux rather than macOS.)

$HOME/sire.app/bin/x86_64-conda_cos6-linux-gnu-g++  test.cpp -I $HOME/sire.app/include/python3.7m -I $HOME/sire.app/include -L $HOME/sire.app/lib $HOME/sire.app/lib/libpython3.7m.a -lrt -lpthread -lm -ldl -lutil -o $HOME/sire.app/bin/test_python

Then move to the install directory:

cd $HOME/sire.app/bin

Launch the interpreter:

./test_python

Import Sire:

import Sire

Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import Sire
[1]    13662 segmentation fault (core dumped)  ./test_python

Previously we fixed this by using a Miniconda that came with a macOS Python interpreter that predated the change to static linking (I think version 3.5). However, it looks like the Python version has got updated along the way (which led to the initial appearance of the segmentation fault) and we have since updated the Miniconda version anyway.

For the source and binary package we could solve this problem by symlinking $HOME/sire.app/bin/python to $HOME/sire.app/sire_python. However, I'm not sure what we can do for the Conda package, other that documenting the workarounds and seeing if the conda-forge folks will remove the static link.

[lohedges]

We've already implemented a supposed solution in the CMakeListst.txt for the wrappers:

if (APPLE)
  # Also need to add '-undefined dynamic_lookup' so that we can build
  # modules that can be imported by a python that has statically linked libPython
  set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -undefined dynamic_lookup" )
endif()

I assume that we are somehow still linking to libpython somehow, or Boost Python is. Anyway, at least we now now what the problem is, i.e. not us. It looks like plenty of other projects, e.g. RDKit, have run across the same issue so I'm sure we can figure out a solution.