Binary build script: Strip Python binary/libraries #1321
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The debug symbols in the Python binaries are as good as unused, and add substantially to the size of the build. Stripping frees up slug space for use by dependencies/the app itself, and also reduces the download/extraction/re-archiving times for both the classic buildpack and CNB. This also matches the approach used by several of our other languages, as well as that for the slim official Python docker image: https://github.com/docker-library/python/blob/1cf43e70e45843c70909a5f914c3c6d0f85fc200/Dockerfile-linux.template#L172 For Python 3.10 (which uses a shared lib), this reduced the build size from 90MB to 55MB. For Python 3.9 (which uses a static lib, and so suffers from the double static lib issue, see: https://bugs.python.org/issue43103) the build size is reduced from 195MB to 69 MB. Figures for Python 3.7/3.8 were not generated, but they will see similar reductions to that for Python 3.9, since they also use the static lib. Python 3.10 diff: ``` $ diff -U0 <(cd shared-lto && du -Sh) <(cd shared-lto-strip/ && du -Sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -1 +1 @@ -56K ./bin +36K ./bin @@ -81 +81 @@ -18M ./lib/python3.10/lib-dynload +5.5M ./lib/python3.10/lib-dynload @@ -103 +103 @@ -26M ./lib +4.1M ./lib ``` ``` $ diff -U0 <(cd shared-lto && du -sh) <(cd shared-lto-strip/ && du -sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -1 +1 @@ -90M . +55M . ``` Python 3.9 diff: ``` $ diff -U0 <(cd 39-orig/ && du -Sh) <(cd 39-strip/ && du -Sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -2 +2 @@ -19M ./lib/python3.9/lib-dynload +5.6M ./lib/python3.9/lib-dynload @@ -91 +91 @@ -56M ./lib/python3.9/config-3.9-x86_64-linux-gnu +7.6M ./lib/python3.9/config-3.9-x86_64-linux-gnu @@ -93,2 +93,2 @@ -56M ./lib -22M ./bin +7.4M ./lib +3.8M ./bin ``` ``` $ diff -U0 <(cd 39-orig/ && du -sh) <(cd 39-strip/ && du -sh) --- /dev/fd/63 +++ /dev/fd/62 @@ -1 +1 @@ -195M . +69M . ``` GUS-W-10989125.
Malax
approved these changes
May 9, 2022
|
For a summary of the combined Python runtime size reductions from this and related PRs, see: |
edmorley
added a commit
that referenced
this pull request
Apr 18, 2024
As part of the CNB multi-architecture support work, we need to change the Python runtime archive S3 URLs to include the architecture name. In addition, for the CNB transition from "stacks" to "targets", it would be helpful to switch from stack ID references (such as `heroku-22`) in the URL scheme, to the distro name+version (eg `ubuntu` and `22.04`) available to CNBs via the CNB targets feature. See: https://github.com/buildpacks/spec/blob/buildpack/0.10/buildpack.md#targets-1 Rather than duplicate the Python archives on S3 under different filenames/locations, it makes sense to migrate this buildpack to the new archive names too, so the same S3 archives can be used by both this buildpack and the CNB. Moving to new archive names/URLs also means we can safely regenerate all existing Python versions to pick up the changes in #1566 (and changes made in the past, such as #1319, #1320, #1321 and #1322), since we won't have to worry about overwriting the old archives (which is something we've typically avoided, since it isn't compatible with the model of being able to roll back to an older buildpack version to return to prior behaviour). Since we're changing the S3 URLs anyway, now is also a good time to make another change that would otherwise cause churn in the S3 URLs again (which affects people that pin buildpack version): Switching archive compression format from gzip to Zstandard (something that we've been wanting to do for a while). Zstandard (aka zstd) is a much superior compression format over gzip (smaller archives and much faster decompression), and is seeing widespread adoption across multiple ecosystems (eg APT packages, Docker images, web browsers etc). See: https://github.com/facebook/zstd https://github.com/facebook/zstd/blob/dev/programs/README.md#usage-of-command-line-interface Our base images already have `zstd` installed (and for Rust for the CNB, there is the [zstd](https://crates.io/crates/zstd) crate available), so it's an easy switch. Various compression levels were tested using zstd's benchmarking feature and in the end the highest level of compression picked, since: 1. Unlike some other compression algorithms, zstd's decompression speed is generally not affected by the compression level. 2. We only have to perform the compression once (when compiling Python). 3. Even at the highest compression ratio, it only takes 20 seconds to compress the Python archives compared to the 10 minutes it takes to compile Python itself (when using PGO+LTO). For the Ubuntu 22.04 Python 3.12.3 archive, switching from gzip to zstd (level 22, with long window mode enabled) results in a 26% reduction in compressed archive size. GUS-W-15158299. GUS-W-15505556.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The debug symbols in the Python binaries are as good as unused, and add substantially to the size of the build. Stripping frees up slug space for use by dependencies/the app itself, and also reduces the download/extraction/re-archiving times for both the classic buildpack and CNB.
This also matches the approach used by several of our other languages, as well as that for the slim official Python docker image:
https://github.com/docker-library/python/blob/1cf43e70e45843c70909a5f914c3c6d0f85fc200/Dockerfile-linux.template#L171-L172
For Python 3.10 (which uses a shared lib), this reduces the build size from 90MB to 55MB.
For Python 3.9 (which uses a static lib, and so suffers from the double static lib issue, see: https://bugs.python.org/issue43103) the build size is reduced from 195MB to 69 MB.
Figures for Python 3.7/3.8 were not generated, but they will see similar reductions to that for Python 3.9, since they also use the static lib.
Python 3.10 diff:
Python 3.9 diff:
See also:
https://www.technovelty.org/linux/stripping-shared-libraries.html
GUS-W-10989125.