armv7 binaries are compiled for armv6

[bnoordhuis]

$ strings node-v8.3.0-linux-armv7l/bin/node | grep arm_version
                 'arm_version': '6',

It's not new, see nodejs/node#4531 (comment).

[gibfahn]

Copying @rvagg 's comment from nodejs/node#4531 (comment)

Thanks for reporting this @igorklopov, I'm a little surprised this went unnoticed for so long. I can confirm your analysis and I've traced it to the fact that we're using Raspbian to get gcc 4.8 on the machine we're using to compile these binaries even though it's a vanilla Debian Wheezy armhf base, and of course Raspbian is armv6 so we get #define __ARM_ARCH 6 from the compiler...

Doh!

Which means we need to go back to finding a Wheezy armhf gcc 4.8(+) source or build one ourselves (which I think I did in the past but that's such a nightmare).

[gibfahn]

@XadillaX are you still working on this (see nodejs/node#4531 (comment))?

[rvagg]

🤦‍♂️ this is technically stalled and it's my fault. I have a server running on scaleway with custom compiled 4.9 gcc packages that I was going to install on those other boxes. I stalled on trying to figure out where to store these packages so they can be reused when reprovisioning the boxes, I don't really want to check them in to this repo.

[piranna]

Don't worry @rvagg, all we have our private lives and sometimes it gets busy :-)

[XadillaX]

@gibfahn Yes but slow progress due to my poor PI3.

[refack]

ping ya'll

[mhdawson]

@rvagg does it make sense to put those on the ci under the download user like we are for packages needed for other platforms ?

[piranna]

@rvagg does it make sense to put those on the ci under the download user like we are for packages needed for other platforms ?

I think it would be a good idea. Would it be possible to also have downloads of binaries compiled with other standard C libraries beside glibc, like musl?

[gibfahn]

@rvagg does it make sense to put those on the ci under the download user like we are for packages needed for other platforms ?

I think it would be a good idea. Would it be possible to also have downloads of binaries compiled with other standard C libraries beside glibc, like musl?

To be clear, @rvagg was talking about putting the gcc versions he compiled himself on https://ci.nodejs.org/downloads/. If you're talking about Node compiled with musl, then that'd be a different thing, and probably worth a separate issue.

Separate question, other than glibc and musl, are there any other C standard libraries in common use?

[piranna]

Separate question, other than glibc and musl, are there any other C standard libraries in common use?

NewLib used to be fairly popular, and although everybody is moving to musl, I think DietLib and uLibC are still popular enough. There are some other ones, but this ones are the first that came to my mind.

[piranna]

For your info, I have at https://github.com/NodeOS/nodejs/releases compilations of Node.js with musl automatically detected, compiled and tested each time a new version is published, if someone is interested :-)

[rvagg]

I uploaded binaries for gcc 4.9.2 armhf here: https://ci.nodejs.org/downloads/armhf/
Built using Jessie source and a debian/control patch I inherited from somewhere, improved for when we had need for it on Wheezy Intel nodes and then modified again for armhf use. (I did this ages ago but they've been sitting on the dedicated Scaleway host I built them on).

4.9.2 is below our recommended minimum still but it works fine, so does 4.8, it's just more likely we'll run into trouble in future (apparently) so we'll just have to deal with that when we get to it and presumably CI will pick up any problems beforehand.

Latest test runs using this new compiler on the 2 test armv7 wheezy hosts are green @ https://ci.nodejs.org/job/node-test-commit-arm/nodes=armv7-wheezy/

So I've gone ahead and upgraded the release host with the same process and started a rebuild of the latest nightly @ https://nodejs.org/download/nightly/v9.0.0-nightly20171016e2015b5347/ (in progress, no cached object files cause of the new compiler so it's quite slow, dates on those files should show when they land).

This also means that any binary produced from ARMv7 from now on will be ARMv7-native, there's potential for breakage here although the only plausible scenario I can think of is that someone is using the ARMv7 binaries on an ARMv6 host and that's easily fixable (i.e. download the ARMv6 binaries instead!).

More info on how this is all set up @ #923

[rvagg]

OK, this is not so great, checking on the build that was produced on an ARMv7 Ubuntu 14.04 machine:

$ ./node-v9.0.0-nightly20171016e2015b5347-linux-armv7l/bin/node -p process.config.variables.arm_version
./node-v9.0.0-nightly20171016e2015b5347-linux-armv7l/bin/node: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./node-v9.0.0-nightly20171016e2015b5347-linux-armv7l/bin/node)

I'm going to have to roll back those changes on the release machine for now until I find a path forward, but it looks like we get locked in to glibc++ 3.4.20+ with this change so we're probably broken for Wheezy and Trusty era Linuxes. That's not a huge drama because Trusty on ARMv7 isn't that prevalent and Raspberry Pi OSs are all Jessie+ based now, but it's still going to break for a lot of folks out there.

[refack]

@rvagg is everything stable for the 9.0.0 release cycle (asking for a friend)?

[rvagg]

back to status quo at least (James' RC0):

sucks to be back at the 6 but I don't have a solution yet so we'll have to roll with that.

[rvagg]

Some more discussion in nodejs/node#16848 related to this, pretty messy. I did some more work on getting a custom 4.9 compiled on Wheezy but the stdlibc++ problem is hard to escape. If V8 forces us to 4.9 then we're going to have to abandon Wheezy.

But perhaps that's not so bad since Wheezy is EOL at the end of May 2018 and Ubuntu 12.04 public LTS is done. Ubuntu 12.04 is in "Extended" till early 2019, that's for paying customers.

We can do this move for Node 10 and that'd work out fine I think for all of this, but the hassle we have now is that we're looking at bringing in V8 6.2 to Node 8 LTS.

[ghost]

Any progress on this?

[rvagg]

OK, so status update on this. I'm afraid that we're probably going to give up on making this work properly for Node 4 - 9 but the good news is that Node 10 is going to come with a new set of minimum supported Linux and compiler versions. This should mean that we get to build Node on Debian Jessie which has newer toolchains available for it.

As of today, I've switched on a new ARMv7 release build configuration that is used for Node 10+, and this includes what's in master today. So nightly versions of Node from now on should be built with a native ARMv7 compiler rather than the one we had to pull in from Raspbian which is for ARMv6+.

The v10.0.0-nightly2018033083d44bee01 nightly has been rebuilt using this new toolchain, so if you grab the ARMv7 build from there you should see the right thing: https://nodejs.org/download/nightly/v10.0.0-nightly2018033083d44bee01/node-v10.0.0-nightly2018033083d44bee01-linux-armv7l.tar.gz

> process.config
...
  variables:
   { arm_float_abi: 'hard',
     arm_fpu: 'vfpv3',
     arm_thumb: 0,
     arm_version: '7',
...
     host_arch: 'arm',
...
     node_byteorder: 'little',
...
     node_tag: '-nightly2018033083d44bee01',
...
     target_arch: 'arm',
...

We haven't locked in final distro, libc and compiler versions for Node 10 but I'm pretty confident this is how we're going to go live with it at this stage.

[piranna]

We haven't locked in final distro, libc and compiler versions for Node 10 but I'm pretty confident this is how we're going to go live with it at this stage.

Could this be able to open the door to have oficial builds using musl in the build matrix?

[rvagg]

@piranna probably not, that's a different question and not dependent on a new major version, it could be done at any time. There's some discussion in #1140, I wouldn't hold my breath but perhaps if there's enough of a call for it and someone can come up with a strategy that doesn't put us on a slippery slow to an explosion in the number of binaries we have to produce then maybe it'll happen.

[piranna]

@piranna probably not, that's a different question and not dependent on a new major version, it could be done at any time

My question was more related to the point that maybe the new toolchain would help about it. Anyway, thank you so much for your detailed explanation :-)

[bnoordhuis]

This has been resolved, AFAICT.