Factor out reproducibility flags for tar and gzip #6884
Conversation
232390b to
926f1e6
Compare
| case "$(uname -s)" in | ||
| Darwin) | ||
| abspath() { python -c 'import os.path, sys; sys.stdout.write(os.path.abspath(sys.argv[1]))' "$@"; } | ||
| canonicalpath() { python -c 'import os.path, sys; sys.stdout.write(os.path.realpath(sys.argv[1]))' "$@"; } | ||
| ;; | ||
| *) | ||
| abspath() { realpath -s "$@"; } | ||
| canonicalpath() { readlink -f "$@"; } | ||
| ;; | ||
| esac |
There was a problem hiding this comment.
Due to NixOS/nixpkgs#94222 readlink -f doesn't work on MacOS within ctx.actions.run.
| # Remove once we upgrade to Bazel >=3.0. Until then `nix-build` output | ||
| # confuses the JAR query in `daml-sdk-head`. | ||
| quiet = True, |
There was a problem hiding this comment.
We are now using Bazel 3.3.1.
| --owner=0 --group=0 --numeric-owner --mtime=2000-01-01\\ 00:00Z --sort=name \\ | ||
| | gzip -n > $@ | ||
| # Avoid file path too long errors on Windows of the form | ||
| # .../tar_dev_env/usr/bin/tar: node_modules/.cache/terser-webpack-plugin/...: file name is too long (cannot be split); not dumped |
There was a problem hiding this comment.
What changed here? Shouldn’t the paths still be the same as before?
There was a problem hiding this comment.
This is a consequence of --format=ustar. Looking at IEEE Std 1003.1-1988 the filename is limited to 256 characters split into two components separated by /: a prefix of at most 155 characters and the filename of at most 100 characters. In this case the filename 97e9576e421fc74fc3b3eed0f6f0ab3be937d0f110bacca2151119236a1a6fc2a5262b044ec008be7300e3489eb49631d1f0572d08ca99384f2f9182ab57 exceeds the 100 character limit.
If need longer file names at some point then reproducible-builds.org recommends
To avoid this, either unset POSIXLY_CORRECT (only works with tar>1.32) or add to the tar call --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime or --format=gnu (both only available in GNU tar) or use --format=ustar if the limitations in that format are not a problem.
to avoid non-deterministic PAX headers. However, it places some restrictions on the tar version.
There was a problem hiding this comment.
Makes sense, thanks for the explanation!
As stated in the comment this is no longer required with Bazel >= 3.0.
This way Bazel will manage the runtime dependencies tar, gzip, mktgz, and patchelf. package-app.sh changes directory so it needs to make sure that all paths are absolute and that the runfiles tree/manifest location is forwarded to programs called by package-app.sh.
changelog_begin changelog_end
The results of
tarandgzipare not reproducible by default due to file ordering or metadata like timestamps, see https://reproducible-builds.org/docs/archives/ for details. These issues can be avoided with the right set of command-line flags totarandgzip. This PR factors these flags out into dedicatedsh_binarytargets that can be invoked bygenrulesand other rules producing archives. This way we don't need to repeat the same set of command line flags over and over and don't run the risk of accidentally forgetting a flag in some place.Additionally,
package-appis converted into ash_binarytarget as well to ensure that runfiles are handled correctly.I have compared the SDK release tarball before and after this change. The only differences were found in
daml-sdk/daml-sdk.jarandstudio/daml-bundled.vsixdue to file ordering and differences in source maps in the navigator bundle. I've manually verified that the navigator and vscode extension work by going through the steps under point 13 inrelease/RELEASE.md. (daml-sdk-headandsdk-version: 0.0.0)Pull Request Checklist
CHANGELOG_BEGINandCHANGELOG_ENDtagsNOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with
/AzurePipelines runtotrigger the build.