OSX compilation with debuginfo isn't deterministic

[alexcrichton]

This is an issue extracted from #47066 (comment) which is caused by an issue that any OSX compilation with debuginfo ends up being nondeterministic. Specifically (currently known at least) the source of nondeterminism is that an mtime for an object file winds up in the final binary.

It turns out this isn't really our fault (unfortunately that makes it harder to fix!). This can be reproduced with just C and a linker:

# Compile an object file with a symbol in it
$ echo 'void foo() {}' > foo.c
$ cc -g foo.c -o foo.o -c

# Link that object to a shared library, take a look at the output
$ cc foo.o -m64 -dynamiclib -o libfoo.dylib -Wl,-dylib
$ md5 libfoo.dylib
MD5 (libfoo.dylib) = e60e735b7c919c19259daddd04a625c8

# update the timestamp on the object file
$ sleep 1
$ touch foo.o

# now link the same way we did above
$ cc foo.o -m64 -dynamiclib -o libfoo.dylib -Wl,-dylib
$ md5 libfoo.dylib
MD5 (libfoo.dylib) = 9754a78562696bbe5912efd9fc892a83

Here we're using the exact same object file (with two timestamps) and we're seeing different linked artifacts.

This is a source of bugs in programs that expect rustc to be deterministic (aka #47066 as was originally stated) and is something that we as rustc should probably fix.

Unfortunately I don't really know of a fix for this myself. I'd be tempted to take a big hammer to the problem and deterministically set all mtime fields for objects going into the linker to a known fixed value, but that unfortunately doesn't fix the determinism for C code (whose objects we don't control) and also is probably too big of a hammer (if a build system uses the mtime of the object to control rebuilds it'd get mixed up).

We could also use something like goblin and reach in to the specific field and remove the actual data. I found it in a symbol section with the N_OSO type (described in various documents online too apparently). We may be able to postprocess all output artifacts on OSX to maybe just zero out these fields unconditionally (or set them to something like May 15, 2015), although I'm not actually sure if this would be easy to do.

[alexcrichton]

cc @michaelwoerister, @johnklai1

cc @luser (you're probably interested in this for the sccache ramifications like @johnklai1 is)

[ranma42]

If we used LLD for linking (#39915), it would be possible to fix this in the linker (by providing a flag to ignore mtime).
This would also fix (this part of) deterministic compilation for other languages as well.

[est31]

Source code for the darwin linker seems to be available, but I have no idea whether they take patches. Maybe LLVM develpers know more. Most likely, Apple will switch to LLD eventually.

[est31]

I wonder what experts on deterministic builds (@infinity0 ) can say about this.

[luser]

Hm. I wonder why we haven't noticed this for Firefox builds? Maybe the version of the linker we're using has a patch to work around this? We're using https://github.com/tpoechtrager/cctools-port for our builds.

[alexcrichton]

@luser that is indeed surprising! The source code there also slurps in the mtime, but that may be getting postprocessed somewhere else perhaps.

[luser]

We discussed this on IRC, and I suspect the reason is that nobody has actually tried to do unstripped reproducible Firefox builds for macOS (although I'm not 100% sure). The info in question are STABS entries used by dsymutil to link the debug info from the object files into the dSYM.

This isn't critical for sccache currently, since it doesn't cache linker outputs.

[luser]

Related: @metajack noticed that static archives are not reproducible on macOS because Apple's ar tool puts timestamps in the archive (mozilla/sccache#169).

[johnklai1]

Right, I think the impact on sccache is similar to mozilla/sccache#169.

What I am seeing is that the .dylib non-determinism is causing unexpected cache misses in sccache since the dylibs end up being passed to rustc.

Here is an example for cargo_metadata:

     Running `/Users/jklai/client-git-2/virtual_env/rust/bin/sccache rustc --crate-name cargo_metadata /Users/jklai/client-git-2/rust/vendor/cargo_metadata-0.2.3/src/lib.rs
--crate-type lib --emit=dep-info,link -C opt-level=1 -C codegen-units=4 -C debuginfo=2 -C debug-assertions=on -C metadata=ef5aea3cb103cbdb -C extra-filename=-ef5aea3cb103cbdb --out-dir /Users/jklai/client-git-2/rust/./target/debug/deps -L dependency=/Users/jklai/client-git-2/rust/./target/debug/deps
--extern serde_derive=/Users/jklai/client-git-2/rust/./target/debug/deps/libserde_derive-b40e6cde3084c26b.dylib
--extern serde=/Users/jklai/client-git-2/rust/./target/debug/deps/libserde-35cb711823dbdd23.rlib
--extern serde_json=/Users/jklai/client-git-2/rust/./target/debug/deps/libserde_json-b77e634c104c128c.rlib --cap-lints allow`

[luser]

Ah, right, proc macro crates!

[DavidGoldman]

This should be fixable by setting the ZERO_AR_DATE=1 in the env of the linker (libtool and ld64). For more context, see this determinism blog post and this commit for Chromium.

I'm not sure of the first Xcode version that supports this though.

[luser]

I'm not sure of the first Xcode version that supports this though.

It should be harmless to set it even if it's not supported, you just won't get a deterministic binary. Here's where it's handled in the most recent ld64 source available, FWIW.