-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build/cmd/relui: ensure fully reproducible builds, including tar.gz/zip archives #24904
Comments
@FiloSottile, were you going to do something here? |
I treated this as lower priority than the changes that would go in the release, but I am going to try to make the final tar.gz/zip reproducible. |
Are we going out of our way to set time stamps in the archives? |
The buildlet sends a tarball to the x/build/cmd/release client. The buildlet uses tar.FileInfoHeader. The source on the buildlet ultimately comes from git (via git archive, via the gitmirror service, via the coordinator), but when the buildlet writes the git archive to disk, it clamps the file time to system time: So it might all work for free today, assuming we have no files with future modtimes in the git repo. But it's a little fragile. It'd be nice if x/build/cmd/release enforced all the modtimes with something predictable, like using the same modtime for all files, picking the time of the git commit of the whole release. |
Change https://go.dev/cl/454836 mentions this issue: |
Change https://go.dev/cl/462198 mentions this issue: |
- Build cmd with CGO_ENABLED=0. Doing so removes the C compiler toolchain from the reproducibility perimeter and also results in cmd/go and cmd/pprof binaries that are statically linked, so that they will run on a wider variety of systems. In particular the Linux versions will run on Alpine and NixOS without needing a simulation of libc.so.6. The potential downside of disabling cgo is that cmd/go and cmd/pprof use the pure Go network resolver instead of the host resolver on Unix systems. This means they will not be able to use non-DNS resolver mechanisms that may be specified in /etc/resolv.conf, such as mDNS. Neither program seems likely to need non-DNS names like those, however. macOS and Windows systems still use the host resolver, which they access without cgo. - Build cmd with -trimpath when building a release. Doing so removes $GOPATH from the file name prefixes stored in the binary, so that the build directory does not leak into the final artifacts. - When CC and CXX are empty, do not pick values to hard-code into the source tree and binaries. Instead, emit code that makes the right decision at runtime. In addition to reproducibility, this makes cross-compiled toolchains work better. A macOS toolchain cross-compiled on Linux will now correctly look for clang, instead of looking for gcc because it was built on Linux. - Convert \ to / in file names stored in .a files. These are converted to / in the final binaries, but the hashes of the .a files affect the final build ID of the binaries. Without this change, builds of a Windows toolchain on Windows and non-Windows machines produce identical binaries except for the input hash part of the build ID. - Due to the conversion of \ to / in .a files, convert back when reading inline bodies on Windows to preserve output file names in error messages. Combined, these four changes (along with Go 1.20's removal of installed pkg/**.a files and conversion of macOS net away from cgo) make the output of make.bash fully reproducible, even when cross-compiling: a released macOS toolchain built on Linux or Windows will contain exactly the same bits as a released macOS toolchain built on macOS. The word "released" in the previous sentence is important. For the build IDs in the binaries to work out the same on both systems, a VERSION file must exist to provide a consistent compiler build ID (instead of using a content hash of the binary). For #24904. Fixes #57007. Change-Id: I665e1ef4ff207d6ff469452347dca5bfc81050e6 Reviewed-on: https://go-review.googlesource.com/c/go/+/454836 Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Russ Cox <rsc@golang.org> Auto-Submit: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
Various Linux distributions edit cmd/go/internal/cfg/cfg.go to change the default settings of GOPROXY and GOSUMDB. Make it possible for them to do this without editing the go command source code by introducing GOROOT/go.env and moving those defaults there. With the upcoming changes for reproducible builds (#24904), this should mean that Linux distributions distribute binaries that are bit-for-bit identical to the Go distribution binaries, even when rebuilding the distribution themselves. Fixes #57179. Change-Id: Ib2ecc61e6d036f97db6fd47dca757c94fdea5629 Reviewed-on: https://go-review.googlesource.com/c/go/+/462198 Auto-Submit: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Russ Cox <rsc@golang.org>
Starting in 1.21 we'll use the |
Change https://go.dev/cl/504117 mentions this issue: |
Change https://go.dev/cl/504116 mentions this issue: |
Change https://go.dev/cl/504057 mentions this issue: |
GOROOT/test is pruned out by cmd/distpack. It isn't really needed for the test anyway; the test can instead use the "src/unicode" subdirectory, which is even within the same module. This test was previously adjusted in CL 13467045 and CL 31859. Unlike in previous iterations of the test, the directories used in this revision are covered by the Go 1 compatibility policy and thus unlikely to disappear. For #24904. Change-Id: I156ae18354bcbc2ddd8d22b210f16ba1e97cd5d1 Reviewed-on: https://go-review.googlesource.com/c/go/+/504116 Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com>
Change https://go.dev/cl/504059 mentions this issue: |
Change https://go.dev/cl/504060 mentions this issue: |
Change https://go.dev/cl/504058 mentions this issue: |
…is missing cmd/distpack removes GOROOT/test, so skip the test if it isn't there. For #24904. Change-Id: Iac381517d0540056b2ccea0dc1bd716113b18468 Reviewed-on: https://go-review.googlesource.com/c/go/+/504117 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com>
cmd/distpack removes GOROOT/test from its archive of the distribution. For #24904. Change-Id: Ifde441f048f8af52f8973555b196ab0520b48ab7 Reviewed-on: https://go-review.googlesource.com/c/go/+/504057 Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com>
cmd/distpack removes GOROOT/test from its distribution. For #24904. Change-Id: I6d1a8c608a1a1fe3fddfe0cd5279202ea9c2b3ce Reviewed-on: https://go-review.googlesource.com/c/go/+/504058 Run-TryBot: Bryan Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
…sn't exist cmd/distpack deletes GOROOT/misc from its distribution. For #24904. Change-Id: I47c60e9a6d39d015683dde7f44bf7c34517b6a8e Reviewed-on: https://go-review.googlesource.com/c/go/+/504059 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com>
cmd/distpack deletes that module from its distribution. For #24904. Change-Id: I69dd328d0f790a49db7a053d703ae985d9ebe9e4 Reviewed-on: https://go-review.googlesource.com/c/go/+/504060 Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com> Auto-Submit: Bryan Mills <bcmills@google.com>
Change https://go.dev/cl/511977 mentions this issue: |
The code was sorting files in the archives entirely by path string, but that's not what fs.WalkDir would do. In a directory with subdirectory foo/bar and file foo/bar.go, foo/bar gets visited first, so foo/bar/baz appears before foo/bar.go, even though "foo/bar/baz" > "foo/bar.go". This CL replaces the string comparison with a path-aware comparison that places foo/bar/baz before foo/bar.go, so that if the tar file is extracted and then repacked using fs.WalkDir, the files will remain in the same order. This will make it easier to compare the pristine distpack-produced tgz for darwin against the rebuilt tgz with signed binaries. Before: % tar tzvf /tmp/cmddist.tgz | grep -C1 runtime/cgo.go -rw-r--r-- 0 0 0 11122 Jul 13 15:00 go/src/runtime/callers_test.go -rw-r--r-- 0 0 0 2416 Jul 13 15:00 go/src/runtime/cgo.go -rw-r--r-- 0 0 0 2795 Jul 13 15:00 go/src/runtime/cgo/abi_amd64.h After: % tar tzvf pkg/distpack/go1.21rsc.src.tar.gz | grep -C1 runtime/cgo.go -rw-r--r-- 0 0 0 1848 Dec 31 1969 go/src/runtime/cgo/signal_ios_arm64.s -rw-r--r-- 0 0 0 2416 Dec 31 1969 go/src/runtime/cgo.go -rw-r--r-- 0 0 0 2479 Dec 31 1969 go/src/runtime/cgo_mmap.go For #24904. For #61513. Change-Id: Ib7374bc0d6324377f81c561bef57fd87b2111b98 Reviewed-on: https://go-review.googlesource.com/c/go/+/511977 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
Change https://go.dev/cl/514898 mentions this issue: |
macOS may write a .DS_Store file to any directory at any time in the general case, unfortunately. Expand test rules to catch them no matter where they may appear. For #24904. Change-Id: I98a2bb5d0ae2ab3a191ff87b33d6f8048ca39aa8 Reviewed-on: https://go-review.googlesource.com/c/go/+/514898 Reviewed-by: Russ Cox <rsc@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Change https://go.dev/cl/560015 mentions this issue: |
As of Go 1.21, the default GOARM value when cross-compiling is 7. This was done in CL 470695. Update its documentation accordingly. For golang/go#24904. For golang/go#65241. Change-Id: I04a22b1e6d5b324615b4822f18db3c53270cf4fb Reviewed-on: https://go-review.googlesource.com/c/website/+/560015 Reviewed-by: Michael Knyszek <mknyszek@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Change https://go.dev/cl/639455 mentions this issue: |
Change https://go.dev/cl/639276 mentions this issue: |
For #24904 For #58884 Change-Id: I0fc61f1a0bc6ebb22bdfd7a7b13ef36c68879384 Reviewed-on: https://go-review.googlesource.com/c/go/+/639276 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Commit-Queue: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
This was changed in CL 470695 but was not mentioned in the release notes. For golang/go#24904 For golang/go#58884 Change-Id: Ib952ce43c65696b3a6091a378587cab1299f30e3 Reviewed-on: https://go-review.googlesource.com/c/website/+/639455 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ian Lance Taylor <iant@golang.org> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
For golang#24904 For golang#58884 Change-Id: I0fc61f1a0bc6ebb22bdfd7a7b13ef36c68879384 Reviewed-on: https://go-review.googlesource.com/c/go/+/639276 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Commit-Queue: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Go already supports reproducible builds (with no action required), but our Go releases have .tar.gz/.zip archives that have timestamps.
We should probably make cmd/release also produce reproducible output and pin the archive file entry timestamps to the git commit time of the tagged commit we're building.
/cc @FiloSottile
The text was updated successfully, but these errors were encountered: