Add Linux loongarch64 architecture support

[zhaixiaojuan]

The LoongArch architecture (LoongArch) is an Instruction Set Architecture (ISA) that has a RISC style.

Documentations:
ISA:
https://loongson.github.io/LoongArch-Documentation/LoongArch-Vol1-EN.html
ABI:
https://loongson.github.io/LoongArch-Documentation/LoongArch-ELF-ABI-EN.html
More docs can be found at:
https://loongson.github.io/LoongArch-Documentation/README-EN.html

[netlify]

✅ Deploy Preview for maturin-guide ready!

Name	Link
🔨 Latest commit	bf0afa5
🔍 Latest deploy log	https://app.netlify.com/sites/maturin-guide/deploys/64806cb075f63400080042ad
😎 Deploy Preview	https://deploy-preview-1521--maturin-guide.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[messense]

Does maturin build and pass tests on loongarch64?

[zhaixiaojuan]

Does maturin build and pass tests on loongarch64?

oh，sorry，maturin depends on target-lexicon which contains architecture related support，we should wait for that first.
I have submitted the relevant pr
#bytecodealliance/target-lexicon#87

[messense]

I think you can update target-lexicon now.

[heiher]

I think you can update target-lexicon now.

Thank you so much for reminding me.

Bump target-lexicon: #1579

[messense]

What's the status now? Does maturin build and pass tests on loongarch64? Should we merge this as-is or wait for something else?

[xiangzhai]

Hi @messense

Does maturin build and pass tests on loongarch64?

cargo test all passed:

    Finished dev [unoptimized + debuginfo] target(s) in 0.37s
    Finished test [unoptimized + debuginfo] target(s) in 0.39s
     Running unittests src/lib.rs (target/debug/deps/maturin-4322abcb583bd9d4)

running 32 tests
test auditwheel::audit::test::test_relpath ... ok
test auditwheel::policy::test::test_policy_musllinux_tag ... ok
test auditwheel::policy::test::test_policy_musllinux_fixup_libc_so_name ... ok
test build_options::test::test_extract_cargo_metadata_args ... ok
test auditwheel::policy::test::test_policy_manylinux_tag ... ok
test auditwheel::policy::test::test_load_policy ... ok
test build_options::test::test_find_bridge_cffi ... ok
test build_options::test::test_find_bridge_pyo3 ... ok
test build_options::test::test_find_bridge_bin ... ok
test build_options::test::test_old_extra_feature_args ... ok
test cargo_toml::test::test_metadata_from_cargo_toml ... ok
test cargo_toml::test::test_metadata_from_cargo_toml_without_authors ... ok
test ci::tests::test_generate_github ... ok
test ci::tests::test_generate_github_abi3 ... ok
test ci::tests::test_generate_github_bin_no_binding ... ok
test ci::tests::test_generate_github_zig_pytest ... ok
test build_context::test::test_macosx_deployment_target ... ok
test build_options::test::test_find_bridge_pyo3_abi3 ... ok
test metadata::test::test_merge_metadata_from_pyproject_dynamic_license_test ... ok
test metadata::test::test_implicit_readme ... ok
test metadata::test::test_merge_metadata_from_pyproject_toml ... ok
test metadata::test::test_path_to_content_type ... ok
test build_options::test::test_find_bridge_pyo3_feature ... ok
test module_writer::tests::sdist_writer_excludes ... ok
test pyproject_toml::tests::deserialize_include_exclude ... ok
test pyproject_toml::tests::test_parse_tool_maturin ... ok
test python_interpreter::config::test::test_load_sysconfig ... ok
test pyproject_toml::tests::test_warn_missing_maturin_version ... ok
test python_interpreter::config::test::test_pyo3_config_file_musl_python_3_11 ... ok
test python_interpreter::config::test::test_pyo3_config_file ... ok
test metadata::test::test_metadata_from_cargo_toml ... ok
test metadata::test::test_merge_metadata_from_pyproject_toml_with_customized_python_source_dir ... ok

test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.55s

     Running unittests src/main.rs (target/debug/deps/maturin-02c12f22342c810f)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/cli.rs (target/debug/deps/cli-6d24acc244d498b3)

running 1 test
Testing tests/cmd/generate-ci.toml ... ok
Testing tests/cmd/maturin.toml ... ok
Testing tests/cmd/init.toml ... ok
Testing tests/cmd/build.toml ... ok
Testing tests/cmd/new.toml ... ok
Testing tests/cmd/develop.toml ... ok
Testing tests/cmd/list-python.toml ... ok
Testing tests/cmd/sdist.toml ... ok
Testing tests/cmd/upload.toml ... ok
Testing tests/cmd/publish.toml ... ok
test cli_tests ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

     Running tests/run.rs (target/debug/deps/run-eea6593603765521)

running 45 tests
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
test abi3_without_version ... ok
test abi3_python_interpreter_args ... ok
Requirement already satisfied: cffi in ./test-crates/venvs/develop-cffi-pure-cpython/lib/python3.10/site-packages (1.15.1)
Requirement already satisfied: pycparser in ./test-crates/venvs/develop-cffi-pure-cpython/lib/python3.10/site-packages (from cffi) (2.21)
Requirement already satisfied: cffi in ./test-crates/venvs/develop-cffi-mixed-cpython/lib/python3.10/site-packages (1.15.1)
Requirement already satisfied: pycparser in ./test-crates/venvs/develop-cffi-mixed-cpython/lib/python3.10/site-packages (from cffi) (2.21)
test develop_hello_world ... ok
test develop_cffi_mixed ... ok
test develop_cffi_pure ... ok
test develop_pyo3_ffi_pure ... ok
Collecting boltons
  Using cached boltons-23.0.0-py2.py3-none-any.whl (194 kB)
Installing collected packages: boltons
Successfully installed boltons-23.0.0
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
test develop_pyo3_mixed_py_subdir ... ok
test develop_pyo3_mixed_include_exclude ... ok
test develop_pyo3_mixed ... ok
test develop_pyo3_pure_conda ... ignored
test develop_uniffi_mixed ... ok
test develop_uniffi_pure ... ok
test develop_pyo3_mixed_src_layout ... ok
    Blocking waiting for file lock on package cache
Ignoring attrs: markers 'extra == "test"' don't match your environment
Ignoring boltons: markers 'sys_platform == "win32" and extra == "test"' don't match your environment
test develop_pyo3_mixed_submodule ... ok
test develop_pyo3_pure ... ok
test integration_hello_world ... ok
test integration_pyo3_bin ... ok
test integration_pyo3_ffi_pure ... ok
test integration_pyo3_mixed_include_exclude ... ok
test integration_pyo3_mixed ... ok
test integration_pyo3_mixed_py_subdir ... ok
test integration_cffi_pure ... ok
test integration_pyo3_mixed_src_layout ... ok
test integration_pyo3_pure_conda ... ok
test integration_uniffi_mixed ... ok
test integration_uniffi_pure ... ok
test integration_cffi_mixed ... ok
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
test invalid_manylinux_does_not_panic ... ok
test lib_with_path_dep_sdist ... ok
test integration_pyo3_mixed_submodule ... ok
    Updating crates.io index
test integration_pyo3_pure ... ok
test musl ... ok
    Blocking waiting for file lock on package cache
error: the lock file /mnt/repo/zhaixiaojuan/maturin/test-crates/lib_with_path_dep/Cargo.lock needs to be updated but --locked was passed to prevent this
If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.
test locked_doesnt_build_without_cargo_lock ... ok
test lib_with_target_path_dep ... ok
    Blocking waiting for file lock on package cache
test pyo3_mixed_include_exclude_sdist ... ok
test pyo3_mixed_src_layout_sdist ... ok
test pyo3_no_extension_module ... ok
test workspace_cargo_lock ... ok
test pyo3_mixed_include_exclude_wheel_files ... ok
test workspace_inheritance_sdist ... ok
test pyo3_source_date_epoch ... ok
test workspace_members_non_local_dep_sdist ... ok
test workspace_sdist ... ok
test workspace_with_path_dep_sdist ... ok
test integration_with_data ... ok

test result: ok. 44 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 47.86s

   Doc-tests maturin

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Thanks,
Leslie Zhai

[messense]

This file this synced with https://github.com/pypa/auditwheel/blob/main/src/auditwheel/policy/manylinux-policy.json

I don't think we should modified it by hand. Loongarch64 doesn't have manylinux support ATM right? why do we need this?

[xiangzhai]

Hi @messense

Thanks for your careful review!

I don't think we should modified it by hand. Loongarch64 doesn't have manylinux support ATM right? why do we need this?

For passing the testcase pyo3_no_extension_module.

I on purpose remove x86_64 in the src/auditwheel/manylinux-policy.json, then it is also able to reproduce the issue for Ubuntu 22.04 x86_64.

I debug the policy_is_satisfied for x86_64 and loongarch64: https://github.com/xiangzhai/maturin/commits/pyo3_no_extension_module

It will early return via AuditWheelError::UnsupportedArchitecture if there was no x86_64 or loongarch64 in the src/auditwheel/manylinux-policy.json:

running 1 test
⚠️  Warning: You're building a library without activating pyo3's `extension-module` feature. See https://pyo3.rs/v0.18.1/building_and_distribution.html#linking
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at /usr/bin/python
⚠️  Warning: Couldn't find the symbol `PyInit_pyo3_no_extension_module` in the native library. Python will fail to import this module. If you're using pyo3, check that `#[pymodule]` uses `pyo3_no_extension_module` as module name
DEBUG: not found arch: x86_64 policy: manylinux_2_5 (aka manylinux1)
DEBUG: not found arch: x86_64 policy: manylinux_2_12 (aka manylinux2010)
DEBUG: not found arch: x86_64 policy: manylinux_2_17 (aka manylinux2014)
DEBUG: not found arch: x86_64 policy: manylinux_2_24
DEBUG: not found arch: x86_64 policy: manylinux_2_27
DEBUG: not found arch: x86_64 policy: manylinux_2_28
DEBUG: not found arch: x86_64 policy: manylinux_2_31
DEBUG: not found arch: x86_64 policy: manylinux_2_34
DEBUG: not found arch: x86_64 policy: manylinux_2_36
DEBUG: not found arch: x86_64 policy: manylinux_2_35
DEBUG: not found arch: x86_64 policy: linux
⚠️  Warning: No compatible platform tag found, using the linux tag instead. You won't be able to upload those wheels to PyPI.
📦 Built wheel for CPython 3.10 to test-crates/targets/pyo3_no_extension_module/pyo3_no_extension_module-2.1.0-cp310-cp310-linux_x86_64.whl
Cause: Should have errored
thread 'pyo3_no_extension_module' panicked at 'Should have errored', tests/common/mod.rs:88:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test pyo3_no_extension_module ... FAILED

failures:

failures:
    pyo3_no_extension_module

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 45 filtered out; finished in 0.49s

Compared with added x86_64:

running 1 test
⚠️  Warning: You're building a library without activating pyo3's `extension-module` feature. See https://pyo3.rs/v0.18.1/building_and_distribution.html#linking
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at /usr/bin/python
⚠️  Warning: Couldn't find the symbol `PyInit_pyo3_no_extension_module` in the native library. Python will fail to import this module. If you're using pyo3, check that `#[pymodule]` uses `pyo3_no_extension_module` as module name
DEBUG: not found arch: x86_64 policy: manylinux_2_5 (aka manylinux1)
DEBUG: not found arch: x86_64 policy: manylinux_2_12 (aka manylinux2010)
DEBUG: not found arch: x86_64 policy: manylinux_2_17 (aka manylinux2014)
DEBUG: not found arch: x86_64 policy: manylinux_2_24
DEBUG: not found arch: x86_64 policy: manylinux_2_27
DEBUG: not found arch: x86_64 policy: manylinux_2_28
DEBUG: not found arch: x86_64 policy: manylinux_2_31
DEBUG: not found arch: x86_64 policy: manylinux_2_34
DEBUG: not found arch: x86_64 policy: manylinux_2_36
=>DEBUG: *found* arch: x86_64 policy: manylinux_2_35
DEBUG: dep: libpython3.10.so.1.0
DEBUG: dep: libgcc_s.so.1
DEBUG: dep: libc.so.6
DEBUG: dep: ld-linux-x86-64.so.2
DEBUG: library: libc.so.6
DEBUG: v: GLIBC_2.2.5
DEBUG: offenders len: 1
test pyo3_no_extension_module ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 45 filtered out; finished in 0.47s

Could you give me some suggestion please?

Thanks,
Leslie Zhai

[messense]

Do you have any instruction for setting up a QEMU environment for loongarch64 on macOS/Linux?

[xiangzhai]

Sorry for my late response! How about https://bbs.loongarch.org/d/123-linux-amd64-qemuloongarch ?

[messense]

That test case should be ignored on platforms that doesn't have manylinux support, I've submitted #1652 to do so.

Can you remove manylinux/musllinux related changes? Thanks!

[messense]

Ah, sorry, pull request from main is difficult to work with, I'll add the commit back in another pr.

#1653