ctz fails with type: nodejs

[Akatuoro]

Running

cds init bookshop-k8s --add tiny-sample,hana,xsuaa,helm,containerize

produces a ctz module

  - name: bookshop-k8s-srv
    build-parameters:
      buildpack:
        type: nodejs
        builder: builder-jammy-base
        path: gen/srv
        env:
          BP_NODE_RUN_SCRIPTS: ""

When running ctz containerize.yaml, building this module fails with

Containerising module bookshop-k8s-srv...
ERROR: failed to build: failed to fetch base layers: saving image with ID "sha256:3274fe28594f484240fc3f17e46b7b45e304830e544dcdd82ce19472d159ef71" from the docker daemon: Error response from daemon: unable to create manifests file: NotFound: content digest sha256:70a3727d92007b1b4d739ebf0060c0f455dca3a95ddd51835fd8331150d2dcb0: not found

Running pack with --buildpack gcr.io/paketo-buildpacks/nodejs manually also fails with the same error. Changing it to --buildpack paketo-buildpacks/nodejs works.

[shubho5]

Hi @Akatuoro,

Can you try updating your pack version to 0.36.4 by doing "brew install buildpacks/tap/pack" (if you are in Mac) and check. Please refer to the following link for more info: https://buildpacks.io/docs/for-platform-operators/how-to/integrate-ci/pack/

Thanks,
Shubho

[Akatuoro]

Hi @shubho5, I was using

pack --version
0.36.4+git-c7f5b1c.build-6274

After a brew reinstall buildpacks/tap/pack, the same error still happens.
The ctz version is 0.2.6.

[Akatuoro]

More logs via ctz -l:

Status: Downloaded newer image for gcr.io/paketo-buildpacks/nodejs:latest
Warning: Builder is trusted but additional modules were added; using the untrusted (5 phases) build flow
0.20.5: Pulling from buildpacksio/lifecycle
Digest: sha256:3274fe28594f484240fc3f17e46b7b45e304830e544dcdd82ce19472d159ef71
Status: Image is up to date for buildpacksio/lifecycle:0.20.5
ERROR: failed to build: failed to fetch base layers: saving image with ID "sha256:3274fe28594f484240fc3f17e46b7b45e304830e544dcdd82ce19472d159ef71" from the docker daemon: Error response from daemon: unable to create manifests file: NotFound: content digest sha256:70a3727d92007b1b4d739ebf0060c0f455dca3a95ddd51835fd8331150d2dcb0: not found

I've also tried to manually install an older version of pack (3.5.1 and 3.6.0), still the exact same error appears

[shubho5]

Can you check in your Docker Desktop -> Settings if "Use containerd for pulling and storing images" is checked or not? Please disable it. Please try to do docker system prune and restart your machine once.

[shubho5]

Please update docker desktop and docker too.

[Akatuoro]

I disabled the containerd image store, now the error is

ERROR: failed to build: downloading buildpack: extracting from registry 'gcr.io/paketo-buildpacks/nodejs': fetching image: image with reference gcr.io/paketo-buildpacks/nodejs:latest was found but its platform (linux) does not match the specified platform (linux/amd64)

Interestingly now the command

pack build <artifactory-registry>/bookshop-k8s:latest --path db --builder paketobuildpacks/builder-jammy-base --buildpack gcr.io/paketo-buildpacks/nodejs -e "BP_NODE_RUN_SCRIPTS=" --publish

works, but (without --publish)

pack build <artifactory-registry>/bookshop-k8s:latest --path db --builder paketobuildpacks/builder-jammy-base --buildpack gcr.io/paketo-buildpacks/nodejs -e "BP_NODE_RUN_SCRIPTS="

throws the same error as ctz.

[Akatuoro]

Just checked what pack does if I don't specify gcr.io as registry for the buildpack.
It has an official registry-index linked in its source. When searching this index for the paketo-buildpacks/nodejs buildpack, it it listed here:
https://github.com/buildpacks/registry-index/blob/b95805aba3f4d768bbf0c3625fd66804a14ac076/no/de/paketo-buildpacks_nodejs
and points to index.docker.io/paketobuildpacks/nodejs, not gcr.io/paketobuildpacks/nodejs.

Using the official registry works, either by specifying paketo-buildpacks/nodejs as buildpack or by omitting it and letting it detect a package-lock.json. This then also works with containerd image pulling.

[shubho5]

Okay. Did you try to delete all the built images and containers and build images freshly? We have seen this issue before and updating docker desktop, pack, disabling "Use containerd for pulling and storing images" solved the issue for us. This is a pack issue to be honest.

Using the official registry works, either by specifying paketo-buildpacks/nodejs as buildpack or by omitting it and letting it detect a package-lock.json. This then also works with containerd image pulling.

I see. You can specify this way to use the official registry image. But we have solved the gcr.io issue using the above mentioned way. I am hoping your docker desktop version is >=4.37.2 and docker version is >=27.4.0

[shubho5]

Are you using intel/Mac machine? FYI you can refer to this issue for more information:
buildpacks/pack#2270

[Akatuoro]

Yes, I'm using a mac with an M4 chip, so apple silicon. I assume that is the problem.

FYI you can refer to this issue for more information:
buildpacks/pack#2270

Also found this, but the example was actually working. That's because it's using the official registry.
So it seems to be an issue with gcr.io + apple silicon (due to containerd image pulling and, if that is disabled, due to the linux vs linux/amd64 platform).

It's not a pack problem, because it only happens when manually specifying gcr.io/paketo-buildpacks/nodejs instead of just paketo-buildpacks/nodejs. The latter is how it is documented in their samples: https://github.com/paketo-buildpacks/samples/tree/main/nodejs/npm

So if there is no specific reason to explicitly have gcr.io as registry, would be good to remove it here:

ctz/lib/parse_yaml.js

Lines 41 to 42 in a5e2956

	if (buildpacks.has(type)) {
	return `gcr.io/paketo-buildpacks/${type}`

[shubho5]

I got your point. But, I didn't see official registry has buildpacks for the other images
https://github.com/SAP/ctz/blob/main/README.md?plain=1#L81-L86

[shubho5]

I found all the official images and raised the PR to use official buildpacks.