Switch to using xz compressed packages?

[chorrell]

I noticed that there are xz packages available. For instances of v4.2.3 (via https://nodejs.org/dist/v4.2.3/)

node-v4.2.3-linux-x64.tar.gz                       03-Dec-2015 22:05            11683079
node-v4.2.3-linux-x64.tar.xz                       03-Dec-2015 22:05             8000024

We've been using the .gz package in all out Dockerfiles. Should we switch to xz? The download would be faster, but I'm not sure what affect it would have on the image build. My understanding of xz is that compression takes a while, but decompression is fast. I'll do a bit of testing this week and see how things play out.

[chorrell]

Did a little experimenting with the recent 5.2.0 image. The build time on my mac is pretty close:

xz:

time docker build 5.2/
Sending build context to Docker daemon 8.192 kB
Step 1 : FROM buildpack-deps:jessie
 ---> 8c1a2b85e52b
Step 2 : RUN set -ex   && for key in     9554F04D7259F04124DE6B476D5A82AC7E37093B     94AE36675C464D64BAFA68DD7434390BDBE9B9C5     0034A06D9D9B0064CE8ADF6BF1747F4AD2306D93     FD3A5288F042B6850C66B31F09FE44734EB7990E     71DCFD284A79C3B38668286BC97EC7A07EDE3FC1     DD8F2338BAE7501E3DD5AC78C273792F7D83545D   ; do     gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$key";   done
 ---> Using cache
 ---> a04526abc207
Step 3 : ENV NPM_CONFIG_LOGLEVEL info
 ---> Using cache
 ---> 445ce4e025fe
Step 4 : ENV NODE_VERSION 5.2.0
 ---> Running in 4843fc11ed98
 ---> dcb8cbcf2176
Removing intermediate container 4843fc11ed98
Step 5 : RUN curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.xz"   && curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc"   && gpg --verify SHASUMS256.txt.asc   && grep " node-v$NODE_VERSION-linux-x64.tar.xz\$" SHASUMS256.txt.asc | sha256sum -c -   && tar -xJf "node-v$NODE_VERSION-linux-x64.tar.xz" -C /usr/local --strip-components=1   && rm "node-v$NODE_VERSION-linux-x64.tar.xz" SHASUMS256.txt.asc
 ---> Running in ac226d3a3c3a
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7965k  100 7965k    0     0  7729k      0  0:00:01  0:00:01 --:--:-- 7741k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3000  100  3000    0     0  22438      0 --:--:-- --:--:-- --:--:-- 22556
gpg: Signature made Wed Dec  9 05:23:57 2015 UTC using RSA key ID 7D83545D
gpg: Good signature from "Rod Vagg <rod@vagg.org>"
gpg:                 aka "Rod Vagg <r@va.gg>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DD8F 2338 BAE7 501E 3DD5  AC78 C273 792F 7D83 545D
node-v5.2.0-linux-x64.tar.xz: OK
 ---> eaf745af8e96
Removing intermediate container ac226d3a3c3a
Step 6 : CMD node
 ---> Running in 534479b37d3a
 ---> cf8b379aafc6
Removing intermediate container 534479b37d3a
Successfully built cf8b379aafc6

real    0m5.184s
user    0m0.174s
sys 0m0.015s

gz:

time docker build 5.2/
Sending build context to Docker daemon 8.192 kB
Step 1 : FROM buildpack-deps:jessie
 ---> 8c1a2b85e52b
Step 2 : RUN set -ex   && for key in     9554F04D7259F04124DE6B476D5A82AC7E37093B     94AE36675C464D64BAFA68DD7434390BDBE9B9C5     0034A06D9D9B0064CE8ADF6BF1747F4AD2306D93     FD3A5288F042B6850C66B31F09FE44734EB7990E     71DCFD284A79C3B38668286BC97EC7A07EDE3FC1     DD8F2338BAE7501E3DD5AC78C273792F7D83545D   ; do     gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$key";   done
 ---> Using cache
 ---> a04526abc207
Step 3 : ENV NPM_CONFIG_LOGLEVEL info
 ---> Using cache
 ---> 445ce4e025fe
Step 4 : ENV NODE_VERSION 5.2.0
 ---> Using cache
 ---> dcb8cbcf2176
Step 5 : RUN curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz"   && curl -SLO "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc"   && gpg --verify SHASUMS256.txt.asc   && grep " node-v$NODE_VERSION-linux-x64.tar.gz\$" SHASUMS256.txt.asc | sha256sum -c -   && tar -xzf "node-v$NODE_VERSION-linux-x64.tar.gz" -C /usr/local --strip-components=1   && rm "node-v$NODE_VERSION-linux-x64.tar.gz" SHASUMS256.txt.asc
 ---> Running in 7679661ec442
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.3M  100 11.3M    0     0  4598k      0  0:00:02  0:00:02 --:--:-- 4598k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3000  100  3000    0     0  16549      0 --:--:-- --:--:-- --:--:-- 16483
gpg: Signature made Wed Dec  9 05:23:57 2015 UTC using RSA key ID 7D83545D
gpg: Good signature from "Rod Vagg <rod@vagg.org>"
gpg:                 aka "Rod Vagg <r@va.gg>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DD8F 2338 BAE7 501E 3DD5  AC78 C273 792F 7D83 545D
node-v5.2.0-linux-x64.tar.gz: OK
 ---> 51b37064e876
Removing intermediate container 7679661ec442
Step 6 : CMD node
 ---> Running in 4641de90ba67
 ---> 4c6176e5f6d6
Removing intermediate container 4641de90ba67
Successfully built 4c6176e5f6d6

real    0m5.845s
user    0m0.170s
sys 0m0.013s

[chorrell]

One thing to note for the above: I already had the base image layers on my mac. The build times on a "clean" docker host would be longer for both.

[chorrell]

Image sizes!

The 5.2.0 image using the current gz compression:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
node                5.2.0               53db8d9b8265        About a minute ago   642.6 MB
buildpack-deps      jessie              a06b823c6a02        12 days ago          606.3 MB

xz compression:

docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
node                5.2.0               a5758a2f0fd8        35 seconds ago      642.6 MB
buildpack-deps      jessie              a06b823c6a02        12 days ago         606.3 MB

Ha! This makes sense though: in both cases it's a single RUN command that gets saved as a new layer. And the final size of node should be the same.

So there could be an improvement to the upstream build time of the image (and the different variants), but it might not be very significant. Probably worth running a few more tests.

[Starefossen]

Thanks for this thorough comparison @chorrell! As you mention this will not have an impact for most users (unless they build the image themselves).

Since xz offers a smaller download and comparable decompression time I am positive to use xz instead of gzip. The only drawback with xz is a higher memory constraint during decompression [1].

[1] Quick Benchmark: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO
[2] Featured on HN: For the love of bits, stop using gzip!

[chorrell]

Thanks!

Yeah, that's pretty much how I lean too. I'll follow up with the upstream Docker people and see what they think.

[chorrell]

A few things to note:

• There are no xz tarballs available for Node.js 0.10.x or 0.12.x
• The slim variant of 5.2.x and 4.2.x (which is based off buildpack-deps:jessie-curl) doesn't come with xz so it would need to be installed via the 'xz-utils' package.

[tianon]

I'm definitely +1 to xz! 😄 FWIW, you can add "xz-utils" to $buildDeps for the slim variants ( https://github.com/nodejs/docker-node/blob/062488d2751e15c34996a2d76112975f39d98ae0/0.10/slim/Dockerfile#L19) and then it'll be removed afterwards too.

[chorrell]

OK cool.

Yeah, unfortunately there are no xz compressed packages for 0.10 and 0.12. We'll need to do something similar for the 4.2.x and 5.3.x images

[Starefossen]

Next patch release of Node.js v0.10 and v0.12 will have xz compressed packages 💯

[chorrell]

Great!