buildkit uses multi-phase build export cache invalidation

[xiangshen123]

buildkit builds in much lower time when using multi-stage builds than when not using multi-stage builds. After I changed dockerfile to multi-stage build in production environment, the build time is three times longer than the original single-stage build time. The specific analysis feels that the export cache is not effective, is there any way to solve this problem

[jedevc]

Can you share the command line you're using? Are you using mode=max in your cache export command?

See https://docs.docker.com/build/cache/backends/#cache-mode.

[xiangshen123]

Can you share the command line you're using? Are you using mode=max in your cache export command?

See https://docs.docker.com/build/cache/backends/#cache-mode.

This is the command I am currently using
buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=${image},push=true,compression=gzip,oci-mediatypes=true

After testing, I looked at the website, made some changes, and tried the following command
uildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --export-cache type=local,dest=/tmp/xs --import-cache type=local,src=/tmp/xs1--output type=image,name=${image},push=true,compression=gzip,oci-mediatypes=true，Found the above problems

[jedevc]

@xiangshen123 you need to include mode=max in your --export-cache argument.

Something like: --export-cache type=local,dest=/tmp/xs,mode=max.

[xiangshen123]

@xiangshen123 you need to include in your argument.mode=max``--export-cache

Something like: .--export-cache type=local,dest=/tmp/xs,mode=max

After listening to your suggestion, I will change the order as follows。
buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --export-cache type=local,dest=/tmp/xs,mode=max --output type=image,name=${image}-nydus,push=true,compression=gzip,oci-mediatypes=true

I did the same build twice, the code did not change at all, according to the actual situation, the second build file did not change, should all the mirror layer will not change, should all go cache, but I found that it did not go cache

After changing to this command, I found that it took me longer to pack. Here are the details of packing

Construct information for the first time：

buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --export-cache type=local,dest=/tmp/xs,mode=max --output type=image,name=xs.linttec.com/linttec-test/content:test-202309271008,push=true,compression=gzip,oci-mediatypes=true
LICENSE? *1 [internal] load .dockerignore
LICENSE? *1 transferring context: 2B done
LICENSE? *1 DONE 0.0s
*2 [internal] load build definition from Dockerfile
*2 transferring dockerfile: 960B done
*2 DONE 0.0s

*3 [auth] public/ibm-semeru-runtimes:pull token for xs.linttec.com
*3 DONE 0.0s

*4 [internal] load metadata for xs.linttec.com/public/ibm-semeru-runtimes:open-17.0.5_8-jre-jammy
*4 DONE 0.2s

*5 [environment 1/5] FROM xs.linttec.com/public/ibm-semeru-runtimes:open-17.0.5_8-jre-jammy@sha256:b6c7864b106564b68b2f53c0f6cdeead3e16933bbc78dfdc43fbbfd5f2b1bc07
*5 resolve xs.linttec.com/public/ibm-semeru-runtimes:open-17.0.5_8-jre-jammy@sha256:b6c7864b106564b68b2f53c0f6cdeead3e16933bbc78dfdc43fbbfd5f2b1bc07 done
*5 DONE 0.0s

*6 [internal] load build context
*6 transferring context: 148.34MB 1.3s done
*6 DONE 1.3s

*7 [builder 5/10] RUN mkdir /app
*7 CACHED

*8 [builder 8/10] COPY extracted/spring-boot-loader/ /app/
*8 CACHED

*9 [builder 2/10] RUN mkdir -p /usr/share/zoneinfo/Asia
*9 CACHED

*10 [builder 6/10] WORKDIR /app
*10 CACHED

*11 [builder 7/10] COPY extracted/dependencies/ /app/
*11 CACHED

*12 [builder 4/10] COPY apiclient_cert.p12 /config/apiclient_cert.p12
*12 CACHED

*13 [builder 3/10] COPY TimeZone /usr/share/zoneinfo/Asia/Shanghai
*13 CACHED

*14 [builder 9/10] COPY extracted/snapshot-dependencies/ /app/
*14 CACHED

*15 [builder 10/10] COPY extracted/application/ /app/
*15 DONE 4.9s

*16 [environment 2/5] COPY --from=builder /usr/share/zoneinfo/Asia/Shanghai /usr/share/zoneinfo/Asia
*16 CACHED

*17 [environment 3/5] RUN ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo Asia/Shanghai > /etc/timezone
*17 CACHED

*18 [environment 4/5] COPY --from=builder /app /app
*18 DONE 0.2s

*19 [environment 5/5] WORKDIR /app
*19 DONE 0.0s

*20 exporting to image
*20 exporting layers
*20 exporting layers 8.3s done
*20 exporting manifest sha256:59104da5be249e14eac426587eb2ee65b79532fde41f68731633614ae0f9c712 done
*20 exporting config sha256:8d7f53860869bc6001ba8bd223a9a373451115f02ade0fec6e967c25ad1e499d done
*20 pushing layers
*20 ...

*21 [auth] linttec-test/content:pull,push token for xs.linttec.com
*21 DONE 0.0s

*20 exporting to image
*20 pushing layers 4.8s done
*20 pushing manifest for xs.linttec.com/linttec-test/content:test-202309271008-nydus@sha256:59104da5be249e14eac426587eb2ee65b79532fde41f68731633614ae0f9c712
*20 pushing manifest for xs.linttec.com/linttec-test/content:test-202309271008-nydus@sha256:59104da5be249e14eac426587eb2ee65b79532fde41f68731633614ae0f9c712 0.4s done
*20 DONE 13.5s

*22 exporting content cache
*22 preparing build cache for export
*22 writing layer sha256:000fe1b92f115d62076e4f9ab08f9ff404f9ba9ae9b8b48c83d7408a28f2a134
*22 writing layer sha256:000fe1b92f115d62076e4f9ab08f9ff404f9ba9ae9b8b48c83d7408a28f2a134 done
*22 writing layer sha256:069c8ad46841329215cd9771d9f08e7df6e291b6bc60abcfa451857b5d212bf3 done
*22 writing layer sha256:0aa8aba22b0d851ac0e1d5b2907acca8ccd00a703465486f9fd90e290b57dc27 done
*22 writing layer sha256:0add4774ba70610a9c8009eb560ee265f25758792722a16edac7956b95ab8489 done
*22 writing layer sha256:20339066af9898fa957759c22854b3a9cfe63f8b2f8b4d30db423ee7bb500506 done
*22 writing layer sha256:37093aae6001873c4e6aab81261ca4a3d240201c4f53dc440047303c652f1786 done
*22 writing layer sha256:470f127844bb9e9599625e0d7fba323b431f68874aba49c30e0ca2b3fb21dba2 done
*22 writing layer sha256:4bb13f403f47fd9b8ed6ef9117592a75e347cc22a7ab844a1a3aeee5ec3e3820 done
*22 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
*22 writing layer sha256:5ffa4b639370af5facf1b2f1a44736634231200d113f1930877a29d16cfdcade done
*22 writing layer sha256:9256f7b1ae62ceaf32dda4bae2045668e86dd9c38de333a0af54a53f74af84a4 done
*22 writing layer sha256:9413f65722b5453be7c9aa9207fe759ffe8c6b150c64cc9915283fba525a976e done
*22 writing layer sha256:a2ff71cdfa5565f2ee4bdacb27a5b43b0936dba2bee1e2eb2ac4ccea75060173
*22 writing layer sha256:a2ff71cdfa5565f2ee4bdacb27a5b43b0936dba2bee1e2eb2ac4ccea75060173 done
*22 writing layer sha256:bb0c16e7199793f1ec6fe8d70dcc9ffbef988fcb30475fa9d73e61bd2160e3f1 done
*22 writing layer sha256:c543ef0e93f0ee5ddf09de34850490f62047138543f9d721c99f9e473e92d527 done
*22 writing layer sha256:c546954cff58a91062ccf09f5df044a530a1dc89e4399b42b1d7703ea615c573 0.0s done
*22 writing layer sha256:d99cd050e17e88857a12d214289ccd5b20e7ba1ea813feb6140d0725acc4be52
*22 preparing build cache for export 9.3s done
*22 writing layer sha256:d99cd050e17e88857a12d214289ccd5b20e7ba1ea813feb6140d0725acc4be52 done
*22 writing layer sha256:e6f6c1f86ee141a8bd1f8a65488a67d89d0424356872cbbb4a3679d7276a43ff done
*22 writing layer sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281 done
*22 writing layer sha256:f122c80ab50734d2c974872f8f7e2c732b98950eef0b867dba7f14d93fdf5e76 done
*22 writing config sha256:ce464cd659b9897cfe6de910d5149d17550ac3c3592a7f016adbbef7849b28d2 done
*22 writing manifest sha256:00f3da7e05b669190e3779f9077d6815301cb1eb8609ced46f8d3fbb26ee099f done
*22 DONE 9.3s

Second build information：
+ buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --export-cache type=local,dest=/tmp/xs,mode=max --output type=image,name=xs.linttec.com/linttec-test/content:test-202309271009,push=true,compression=gzip,oci-mediatypes=true
*1 [internal] load .dockerignore
*1 transferring context: 2B done
*1 DONE 0.0s

*2 [internal] load build definition from Dockerfile
*2 transferring dockerfile: 960B done
*2 DONE 0.0s

*3 [internal] load metadata for xs.linttec.com/public/ibm-semeru-runtimes:open-17.0.5_8-jre-jammy
*3 DONE 0.0s

*4 [builder 1/10] FROM xs.linttec.com/public/ibm-semeru-runtimes:open-17.0.5_8-jre-jammy@sha256:b6c7864b106564b68b2f53c0f6cdeead3e16933bbc78dfdc43fbbfd5f2b1bc07
*4 resolve xs.linttec.com/public/ibm-semeru-runtimes:open-17.0.5_8-jre-jammy@sha256:b6c7864b106564b68b2f53c0f6cdeead3e16933bbc78dfdc43fbbfd5f2b1bc07 done
*4 DONE 0.0s

*5 [internal] load build context
*5 transferring context: 3.00MB 0.1s done
*5 DONE 0.1s

*6 [builder 2/10] RUN mkdir -p /usr/share/zoneinfo/Asia
*6 CACHED

*7 [builder 5/10] RUN mkdir /app
*7 CACHED

*8 [builder 3/10] COPY TimeZone /usr/share/zoneinfo/Asia/Shanghai
*8 CACHED

*9 [builder 4/10] COPY apiclient_cert.p12 /config/apiclient_cert.p12
*9 CACHED

*10 [builder 6/10] WORKDIR /app
*10 CACHED

*11 [builder 7/10] COPY extracted/dependencies/ /app/
*11 CACHED

*12 [builder 8/10] COPY extracted/spring-boot-loader/ /app/
*12 CACHED

*13 [builder 9/10] COPY extracted/snapshot-dependencies/ /app/
*13 CACHED

*14 [builder 10/10] COPY extracted/application/ /app/
*14 DONE 3.6s

*15 [environment 2/5] COPY --from=builder /usr/share/zoneinfo/Asia/Shanghai /usr/share/zoneinfo/Asia
*15 CACHED

*16 [environment 3/5] RUN ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo Asia/Shanghai > /etc/timezone
*16 CACHED

*17 [environment 4/5] COPY --from=builder /app /app
*17 DONE 0.2s

*18 [environment 5/5] WORKDIR /app
*18 DONE 0.0s

*19 exporting to image
*19 exporting layers
*19 exporting layers 8.6s done
*19 exporting manifest sha256:1c6fbf476907bc232c9cc126529e4e68da23b28227ac1c4690148c48f59e382d done
*19 exporting config sha256:783970ab81087eaa05371bc4a296f0115ba4fde0bbb9b6d8fea0fb76f537810b done
*19 pushing layers
*19 pushing layers 4.8s done
*19 pushing manifest for xs.linttec.com/linttec-test/content:test-202309271009-nydus@sha256:1c6fbf476907bc232c9cc126529e4e68da23b28227ac1c4690148c48f59e382d
*19 pushing manifest for xs.linttec.com/linttec-test/content:test-202309271009-nydus@sha256:1c6fbf476907bc232c9cc126529e4e68da23b28227ac1c4690148c48f59e382d 0.2s done
*19 DONE 13.5s

*20 exporting content cache
*20 preparing build cache for export
*20 writing layer sha256:000fe1b92f115d62076e4f9ab08f9ff404f9ba9ae9b8b48c83d7408a28f2a134
*20 writing layer sha256:000fe1b92f115d62076e4f9ab08f9ff404f9ba9ae9b8b48c83d7408a28f2a134 done
*20 writing layer sha256:069c8ad46841329215cd9771d9f08e7df6e291b6bc60abcfa451857b5d212bf3 done
*20 writing layer sha256:0aa8aba22b0d851ac0e1d5b2907acca8ccd00a703465486f9fd90e290b57dc27 done
*20 writing layer sha256:0add4774ba70610a9c8009eb560ee265f25758792722a16edac7956b95ab8489 done
*20 writing layer sha256:20339066af9898fa957759c22854b3a9cfe63f8b2f8b4d30db423ee7bb500506 done
*20 writing layer sha256:37093aae6001873c4e6aab81261ca4a3d240201c4f53dc440047303c652f1786 done
*20 writing layer sha256:46c3cc8c1b40965dc19ca37501b971d2be15158b1e502cada1b19af0522777bd
*20 writing layer sha256:46c3cc8c1b40965dc19ca37501b971d2be15158b1e502cada1b19af0522777bd done
*20 writing layer sha256:470f127844bb9e9599625e0d7fba323b431f68874aba49c30e0ca2b3fb21dba2 done
*20 writing layer sha256:4bb13f403f47fd9b8ed6ef9117592a75e347cc22a7ab844a1a3aeee5ec3e3820 done
*20 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
*20 writing layer sha256:5fcf7b0707186a8041de7c20a98238ef8645a8ff0229cf97e1018420cf6bae1c 0.0s done
*20 writing layer sha256:5ffa4b639370af5facf1b2f1a44736634231200d113f1930877a29d16cfdcade
*20 preparing build cache for export 8.4s done
*20 writing layer sha256:5ffa4b639370af5facf1b2f1a44736634231200d113f1930877a29d16cfdcade done
*20 writing layer sha256:9256f7b1ae62ceaf32dda4bae2045668e86dd9c38de333a0af54a53f74af84a4 done
*20 writing layer sha256:9413f65722b5453be7c9aa9207fe759ffe8c6b150c64cc9915283fba525a976e done
*20 writing layer sha256:bb0c16e7199793f1ec6fe8d70dcc9ffbef988fcb30475fa9d73e61bd2160e3f1 done
*20 writing layer sha256:c543ef0e93f0ee5ddf09de34850490f62047138543f9d721c99f9e473e92d527 done
*20 writing layer sha256:d99cd050e17e88857a12d214289ccd5b20e7ba1ea813feb6140d0725acc4be52 done
*20 writing layer sha256:e6f6c1f86ee141a8bd1f8a65488a67d89d0424356872cbbb4a3679d7276a43ff done
*20 writing layer sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281 done
*20 writing layer sha256:f122c80ab50734d2c974872f8f7e2c732b98950eef0b867dba7f14d93fdf5e76 done
*20 writing config sha256:868fc9515f603fbd5de6cfca007387099178ec1eb5319bfb213c6aaf75855789 done
*20 writing manifest sha256:1f45ecce66ef5d923dbc1682f2835bfdf03501e80ee3bf6ec724636d75ef05b9 done
*20 DONE 8.4s

I wonder why it takes so long to build and it doesn't feel like the cache is working

[xiangshen123]

您需要包含在您的论点中。mode=max``--export-cache

像这样： .--export-cache type=local,dest=/tmp/xs,mode=max

Can you help me see exactly why that is

[jedevc]

From the output you share, it looks like the cache is working - "CACHED" is in the output, for each step, so the cache is working.

The export step is taking more time now, since there is more cache to export - this is the disadvantage of mode=max.

I'm going to close this issue, since it looks like this is resolved - feel free to keep discussing though 🎉