Polars does not build on Apple M1/2

[orlp]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Reproducible example (on an Apple M1/2)

git clone git@github.com:pola-rs/polars.git
cd polars
cargo build --release

Issue description

The error happens during linking

error: linking with `cc` failed: exit status: 1
= note: LC_ALL="C"
<... snip gigantic command ...>
  = note: ld: warning: search path '/usr/local/lib' not found
          ld: warning: no platform load command found in '/private/var/folders/90/mrbxkzl12756q0n_vff7sqjc0000gn/T/rustcGgCQK9/symbols.o', assuming: macOS
          ld: Undefined symbols:
            _PyBytes_AsString, referenced from:
                pyo3::types::string::PyString::to_string_lossy::h33fe334d6a9e1270 in libpyo3-efd819b20aaf5f51.rlib[17](pyo3-efd819b20aaf5f51.pyo3.29836c1fadb8fd63-cgu.14.rcgu
.o)
<... snip a lot of repetitive undefined Python symbols ...>

clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: could not compile `python_rust_compiled_function` (lib) due to previous error

I believe the problem is related to PyO3/pyo3#1800. PyO3 says that on MacOS some extra linking options need to be set: https://pyo3.rs/v0.19.2/building_and_distribution#macos . Namely

[target.x86_64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

[target.aarch64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

They also say that this may be necessary

[build]
rustflags = [
  "-C", "link-args=-Wl,-rpath,/Library/Developer/CommandLineTools/Library/Frameworks",
]

Expected behavior

I expect Polars to build out-of-the-box.

Installed versions

No features.

[orlp]

The problem is setting these flags only for the python_rust_compiled_function crate.

https://doc.rust-lang.org/cargo/reference/config.html:

At present, when being invoked from a workspace, Cargo does not read config files from crates within the workspace. i.e. if a workspace has two crates in it, named /projects/foo/bar/baz/mylib and /projects/foo/bar/baz/mybin, and there are Cargo configs at /projects/foo/bar/baz/mylib/.cargo/config.toml and /projects/foo/bar/baz/mybin/.cargo/config.toml, Cargo does not read those configuration files if it is invoked from the workspace root (/projects/foo/bar/baz/).

I don't exactly like setting those PyO3 flags on the entire polars project on MacOS.

[stinodego]

I believe this is the culprit:
https://pyo3.rs/main/faq#i-cant-run-cargo-test-or-i-cant-build-in-a-cargo-workspace-im-having-linker-issues-like-symbol-not-found-or-undefined-reference-to-_pyexc_systemerror

We should probably remove that package from the default workspace, or configure it a bit differently. It's not just a problem on MacOS, it also blocks running cargo test on the workspace currently.

Not 100% why we need the examples crates at all. Can't we replace those with proper Cargo examples?

[orlp]

Actually, I managed to make it work as it is right now (without setting a flag on the entire workspace) by the alternative proposed by PyO3. I added a build.rs script to the example crate only, and that build.rs script contains

fn main() {
    pyo3_build_config::add_extension_module_link_args();
}

which sets the appropriate rustflags, only for that crate. I will make a pull request for this.

[orlp]

We should probably remove that package from the default workspace, or configure it a bit differently. It's not just a problem on MacOS, it also blocks running cargo test on the workspace currently.

@stinodego I will leave those decisions to you, perhaps you should make a new issue for this. I just want to make the smallest possible change to make polars compile out-of-the-box on MacOS right now.

[stinodego]

We should probably remove that package from the default workspace, or configure it a bit differently. It's not just a problem on MacOS, it also blocks running cargo test on the workspace currently.

@stinodego I will leave those decisions to you, perhaps you should make a new issue for this. I just want to make the smallest possible change to make polars compile out-of-the-box on MacOS right now.

Right, I thought these were related but they're not! Thanks for the fix.