Cross-compile to Android does not work

[AxelNennker]

I could not get this crate to work for cargo build --target=aarch64-linux-android

Even if I tell cargo which compilers to use. The following does not work:
ANDROID_API=28 PATH=$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin:$PATH CC=$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang CXX=$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang++ cargo build --target=aarch64-linux-android

The value for $NDK is set using

export HOST_TAG=linux-x86_64
export NDK=$HOME/Android/Sdk/ndk-bundle

I suggest to use zeromq's own build system in this crate, that is basically
cd vendor && ./autogen.sh && HOST_TAG=linux-x86_64 NDK=$HOME/Android/Sdk/ndk-bundle LIBZMQ_SYS_STATIC=1 PATH=$NDK/toolchains/llvm/prebuilt/HOST_TAG/bin:$PATH CC=$NDK/toolchains/llvm/prebuilt/$HOST_TAG/bin/aarch64-linux-android28-clang CXX=$NDK/toolchains/llvm/prebuilt/$HOST_TAG/bin/aarch64-linux-android28-clang++ ./configure --target= --host=x86_64-linux-gnu --enable-static --disable-shared --prefix=/usr/local/aarch64-linux-android && make

If you are not on Linux you have to use the correct HOST_TAG for your system.
Just see what is there after installing Android Studio.

I wouldn't mind you get cmake do the job but a PR doing this with ./configure is #3

[jean-airoldie]

Could you submit the stacktrace that you get when trying to compile using cmake?

I'm not sure I understand your point about using zeromq's build system. What would be the advantage of using your method vs. statically cross-compiling libzmq outside of cargo (e.g. using autogen & make) then telling cargo where the compiled lib is?

[AxelNennker]

I am building a library for Android that uses zeromq.
https://github.com/hyperledger/indy-sdk/tree/master/libindy
Currently I have to compile several libraries and then tell that cargo where the three other libraries are.
openssl, libsodium, zeromq

We have built some scripts that somewhat automate the process but currently we are stuck with NDK r16

To simplify the process I would like that these other libraries are build using cargo.
So in libindy I like to call cargo build --target=aarch64-linux-android and want that the crates used by libindy magically compile for this target.

This seems to work for openssl (feature vendored) and for sodiumoxide but not for zmq.

• sodiumoxide

ignisvulpis@namenlos:~/development/sodiumoxide$ HOST_TAG=linux-x86_64 LIBZMQ_SYS_STATIC=1 PATH=$NDK/toolchains/llvm/prebuilt/HOST_TAG/bin:$PATH CC=$NDK/toolchains/llvm/prebuilt/$HOST_TAG/bin/aarch64-linux-android28-clang CXX=$NDK/toolchains/llvm/prebuilt/$HOST_TAG/bin/aarch64-linux-android28-clang++ cargo build --target=aarch64-linux-android 
    Updating crates.io index
   Compiling libc v0.2.54
   Compiling byteorder v1.3.1
   Compiling crc32fast v1.2.0
   Compiling cfg-if v0.1.7
   Compiling adler32 v1.0.3
   Compiling pkg-config v0.3.14
   Compiling cc v1.0.36
   Compiling serde v1.0.90
   Compiling filetime v0.2.5
   Compiling xattr v0.2.2
   Compiling libflate v0.1.22
   Compiling tar v0.4.24
   Compiling libsodium-sys v0.2.2 (/home/ignisvulpis/development/sodiumoxide/libsodium-sys)
   Compiling sodiumoxide v0.2.2 (/home/ignisvulpis/development/sodiumoxide)
    Finished dev [unoptimized + debuginfo] target(s) in 31.42s
ignisvulpis@namenlos:~/development/sodiumoxide$ 

Regarding the stacktrace: I seems that zeromq-src-rs is not building any library. Maybe I am using it wrong.

ignisvulpis@namenlos:/tmp/zeromq-src-rs$ cargo clean && cargo build --target=aarch64-linux-android
   Compiling cc v1.0.37
   Compiling cmake v0.1.40
   Compiling zeromq-src v0.1.1 (/tmp/zeromq-src-rs)
    Finished dev [unoptimized + debuginfo] target(s) in 1.62s
ignisvulpis@namenlos:/tmp/zeromq-src-rs$ find target -name lib\*
target/aarch64-linux-android/debug/libzeromq_src.rlib
target/aarch64-linux-android/debug/deps/libcmake-9820f53154040bb2.rlib
target/aarch64-linux-android/debug/deps/libzeromq_src-08db56fb795a94e1.rlib
target/aarch64-linux-android/debug/deps/libcc-6254c2c529020aab.rlib
target/aarch64-linux-android/debug/.fingerprint/cmake-9820f53154040bb2/lib-cmake-9820f53154040bb2.json
target/aarch64-linux-android/debug/.fingerprint/cmake-9820f53154040bb2/lib-cmake-9820f53154040bb2
target/aarch64-linux-android/debug/.fingerprint/cc-6254c2c529020aab/lib-cc-6254c2c529020aab
target/aarch64-linux-android/debug/.fingerprint/cc-6254c2c529020aab/lib-cc-6254c2c529020aab.json
target/aarch64-linux-android/debug/.fingerprint/zeromq-src-08db56fb795a94e1/lib-zeromq_src-08db56fb795a94e1.json
target/aarch64-linux-android/debug/.fingerprint/zeromq-src-08db56fb795a94e1/lib-zeromq_src-08db56fb795a94e1
target/aarch64-linux-android/debug/libzeromq_src.d
ignisvulpis@namenlos:/tmp/zeromq-src-rs$ 

There is no libzmq.a nor a libzmq.so

[jean-airoldie]

Understood.

Can you run cargo test --all --target=aarch64-linux-android? If the test does not fail then the lib has indeed compiled. In itself the crate does not compile the lib, it merely gives the tools required to do so. But running the tests will compile the testcrate which calls these tools to compile, link and run a test script.

[AxelNennker]

The first naive try does not find the compiler.

ignisvulpis@namenlos:/tmp/zeromq-src-rs$ cargo clean && cargo test --all --target=aarch64-linux-android
   Compiling cc v1.0.37
   Compiling cmake v0.1.40
   Compiling zeromq-src v0.1.1 (/tmp/zeromq-src-rs)
   Compiling testcrate v0.1.0 (/tmp/zeromq-src-rs/testcrate)
error: failed to run custom build command for `testcrate v0.1.0 (/tmp/zeromq-src-rs/testcrate)`
process didn't exit successfully: `/tmp/zeromq-src-rs/target/debug/build/testcrate-7a6d0b4f4a531c39/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-changed=build.rs
running: "cmake" "/tmp/zeromq-src-rs/vendor" "-DENABLE_DRAFTS=OFF" "-DCMAKE_BUILD_TYPE=Release" "-DWITH_PERF_TOOL=OFF" "-DBUILD_SHARED=OFF" "-DBUILD_STATIC=ON" "-DCMAKE_INSTALL_PREFIX=/tmp/zeromq-src-rs/target/aarch64-linux-android/debug/build/testcrate-0f017105baa04b79/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC --target=aarch64-linux-android" "-DCMAKE_C_COMPILER=aarch64-linux-android-clang" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC --target=aarch64-linux-android" "-DCMAKE_CXX_COMPILER=aarch64-linux-android-clang++"
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
-- Configuring incomplete, errors occurred!
See also "/tmp/zeromq-src-rs/target/aarch64-linux-android/debug/build/testcrate-0f017105baa04b79/out/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/zeromq-src-rs/target/aarch64-linux-android/debug/build/testcrate-0f017105baa04b79/out/build/CMakeFiles/CMakeError.log".

--- stderr
CMake Error at CMakeLists.txt:2 (project):
  The CMAKE_C_COMPILER:

    aarch64-linux-android-clang

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
  the compiler, or to the compiler name if it is in the PATH.


CMake Error at CMakeLists.txt:2 (project):
  The CMAKE_CXX_COMPILER:

    aarch64-linux-android-clang++

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
  to the compiler, or to the compiler name if it is in the PATH.


thread 'main' panicked at '
command did not execute successfully, got: exit code: 1

build script failed, must exit now', /home/ignisvulpis/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.40/src/lib.rs:832:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

ignisvulpis@namenlos:/tmp/zeromq-src-rs$ 

and cmake suggest the 'wrong' compiler. Current Android clang compilers have the Android-API in their name.
So setting the environment as suggested by the error message.

ignisvulpis@namenlos:/tmp/zeromq-src-rs$ cargo clean && ANDROID_API=28 HOST_TAG=linux-x86_64 NDK=$HOME/Android/Sdk/ndk-bundle CC=$NDK/toolchains/llvm/prebuilt/$HOST_TAG/bin/aarch64-linux-android28-clang CXX=$NDK/toolchains/llvm/prebuilt/$HOST_TAG/bin/aarch64-linux-android28-clang++ cargo test --all --target=aarch64-linux-android

cmake-output.txt

[jean-airoldie]

From what I understand cmake is running a test to check if the supplied compilers are working. But since you are cross compiling you can't actually run an executable on host.

From what I understand the solution would be to disable these tests. We can do so by passing CMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY. I will make a PR to add a cross-compiling method to the Build that will set this define.

[AxelNennker]

I think the linker is failing because it is not finding some file like e.g. crtend_android.o
Most of the times this means that sysroot is not found.
Don't know why cmake fails to find them. Usually it is the job of the compiler or linker to find them.
I tried to add --sysroot to CFLAGS but no success

[jean-airoldie]

I added the ability to pass abitrary cmake definitions in this WIP PR #5. I'm passing CMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY.

Please run cargo test --all against this PR and tell me if it helps.

edit: Nevermind I forgot to cargo clean, this PR does not even compile.

[AxelNennker]

Please see attached file for the output. No change, I think.

cmake-cross-compilation.txt

[jean-airoldie]

No change indeed.

Sorry I can't help much. This seems to be an issue related to the cmake configuration and I don't have any experience using cmake in a cross-compiling context.

[AxelNennker]

Then maybe use my PR which doesn't use cmake but configure && make?

[jean-airoldie]

I don't really think it makes sense to merge that PR considering it doesn't have any proper linking logic (which is why it doesn't compile in the first place). The purpose of this crate is to be able to use libzmq without having to install / compile it as a dependency. It lets cargo handle the compilation and linking. I am not against using configure && make but they are not well integrated into the rust ecosystem (properly linking for every system would be a real headache). If I merge your PR, this repo would basically be the equivalent of a sh script with a git submodule that compiles the lib. I don't think it would be particularly reliable.

Does that make sense?

[AxelNennker]

I agree on the purpose of this crate. It should be integrated in libzmq-rs and rust-zmq behind a feature flag 'vendored'

I think it a good idea to follow the example of the openssl source crate openssl-src
https://github.com/alexcrichton/openssl-src-rs/blob/master/src/lib.rs#L80

Which calls openssl's ./Configure script.

I think the PR brings this crate forward.
The PR does not finish the work to integrate the crate into libzmqrs or rust-zmq.

Regarding reliability, I think that ./Configure is THE way to build openssl from source.

[jean-airoldie]

Yes I don't disagree with using configure and make, I'm more worried about the linking procedure. As you can see in the crate that you link, making it work for multiple arch seems like a real headache.

The reason openssl is not build using cmake is because there is no support for it (i.e. no `CMakeLists.txt anywhere in the project).

I can't really merge the PR as is since it as no linking logic currently. But I saw that it is now a WIP.

[oblique]

I can compile for Android using cmake without any problem. These are the full steps:

Download and unzip Android NDK.

Now run:

cd android-ndk-r19c
./build/tools/make-standalone-toolchain.sh --arch=arm64 --platform=android-28 --install-dir=android-toolchain
export PATH="$PATH:$PWD/android-toolchain/bin"
cd /path/to/zeromq-src-rs
cargo build --all --target=aarch64-linux-android

So the key to this is to use make-standalone-toolchain.sh to generate the toolchain in a directory.

[AxelNennker]

Works for me too. Thanks for checking this out. I thought that make-standalone-toolchain.sh was deprecated by Google/Android.
Anyway, great this works.

[oblique]

I manage to build it without standalone toolchain.

Create android.cmake and put the following in:

set(ANDROID_PLATFORM android-28)
set(ANDROID_ABI arm64-v8a)
string(REPLACE "--target=aarch64-linux-android" "" CMAKE_C_FLAGS "${CMAKE_C_FLAGS}")
string(REPLACE "--target=aarch64-linux-android" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
unset(CMAKE_C_COMPILER CACHE)
unset(CMAKE_CXX_COMPILER CACHE)
include("$ENV{ANDROID_NDK_PATH}/build/cmake/android.toolchain.cmake")

Now you can build with:

export ANDROID_NDK_PATH=/path/to/android-ndk-r19c
export CMAKE_TOOLCHAIN_FILE=/path/to/android.cmake
cargo build --target=aarch64-linux-android --all

Of-course we can avoid all the above by improving cmake-rs.

[AxelNennker]

There is an issue for that here: rust-lang/cmake-rs#80