Docker image build failing

[dyladan]

I didn't know how else to reach out about this issue but we're having an issue building the docker image. On step 19/39 of the docker build command we are getting this error:

[2021-10-28T13:51:02.210Z] Step 19/39 : RUN make amd64-build
[2021-10-28T13:51:02.210Z]  ---> Running in 326b228d4493
[2021-10-28T13:51:02.210Z] make: cat: Operation not permitted
[2021-10-28T13:51:02.210Z] make: /bin/sh: Operation not permitted
[2021-10-28T13:51:02.210Z] make: /bin/sh: Operation not permitted
[2021-10-28T13:51:02.210Z] make: go: Operation not permitted
[2021-10-28T13:51:02.210Z] make: /bin/sh: Operation not permitted
[2021-10-28T13:51:02.210Z] cd tools/linters && GOBIN=/workspace/bin go install golang.org/x/tools/cmd/goimports
[2021-10-28T13:51:02.210Z] make: /bin/sh: Operation not permitted
[2021-10-28T13:51:02.210Z] make: *** [Makefile:135: install-tools] Error 127
[2021-10-28T13:51:02.210Z] The command '/bin/sh -c make amd64-build' returned a non-zero code: 2

The command used to build the docker image is this:

docker build -t ${IMAGE_NAME}:${currentBuild.number} --file ./aws-otel-collector/cmd/awscollector/Dockerfile aws-otel-collector

This command has always worked in the past for previous builds until this week. Unfortunately I don't have the exact commit sha of the most recent successful and the first failed builds, but the last success was 7 days ago on October 22nd at 2AM UTC and the first failed build was 6 days ago on October 23rd at 2AM UTC. Since then we have not had a successful build.

[sethAmazon]

What os are you running this on?

[sethAmazon]

Is there a reason you do not want to take the latest version from ecr? Are you actively developing for this repo?

[sethAmazon]

What cpu arc do you have?

[sethAmazon]

What is you output when you run make docker-build

[anuraaga]

@dyladan Sorry if it sounds too tech-supporty but have you tried restarting your docker daemon? I can't remember if it was operation not permitted but do occasionally get mysterious errors that get solved with a restart.

[dyladan]

this is running in CI on short lived machines so yes it has been restarted :)

Tried building on both my local machine (MacOS 11.6 on Intel MBP 16") and on our CI servers which run some flavor of linux, not sure exactly which. Same error in both places.

Is there a reason you do not want to take the latest version from ecr? Are you actively developing for this repo?

As mentioned in #516 I maintain the Dynatrace exporter for the collector and we test all supported versions nightly in order to ensure no changes on our side have broken anything and that no changes on the collector have broken anything. Testing the latest main branch helps us catch any potential problems before they're actually released.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[alolita]

Resolved based on response from @dyladan

[sethAmazon]

Take a look at this repo public.ecr.aws/aws-otel-test/adot-collector-integration-test

[dyladan]

This is not resolved

[dyladan]

Log output from this morning trying to build the latest collector in CI:

[2022-01-07T16:03:09.862Z] + docker build -t aws-otel-collector-exporter-tests-collector-image:2 --file ./aws-otel-collector/cmd/awscollector/Dockerfile aws-otel-collector

[2022-01-07T16:03:10.161Z] Sending build context to Docker daemon  3.208MB


[2022-01-07T16:03:10.420Z] Step 1/37 : FROM alpine:latest as build

[2022-01-07T16:03:11.863Z] latest: Pulling from library/alpine

[2022-01-07T16:03:11.863Z] 59bf1c3509f3: Pulling fs layer

[2022-01-07T16:03:11.863Z] 59bf1c3509f3: Verifying Checksum

[2022-01-07T16:03:11.863Z] 59bf1c3509f3: Download complete

[2022-01-07T16:03:12.470Z] 59bf1c3509f3: Pull complete

[2022-01-07T16:03:12.470Z] Digest: sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300

[2022-01-07T16:03:12.470Z] Status: Downloaded newer image for alpine:latest

[2022-01-07T16:03:12.470Z]  ---> c059bfaa849c

[2022-01-07T16:03:12.470Z] Step 2/37 : ENV GOPROXY=direct

[2022-01-07T16:03:12.470Z]  ---> Running in fe70c71546d3

[2022-01-07T16:03:12.728Z] Removing intermediate container fe70c71546d3

[2022-01-07T16:03:12.728Z]  ---> 9e75086431f9

[2022-01-07T16:03:12.728Z] Step 3/37 : RUN apk --update add ca-certificates

[2022-01-07T16:03:12.728Z]  ---> Running in 044ba89f7e21

[2022-01-07T16:03:13.013Z] fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz

[2022-01-07T16:03:13.581Z] fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz

[2022-01-07T16:03:13.839Z] (1/1) Installing ca-certificates (20191127-r7)

[2022-01-07T16:03:13.839Z] Executing busybox-1.34.1-r3.trigger

[2022-01-07T16:03:13.839Z] Executing ca-certificates-20191127-r7.trigger

[2022-01-07T16:03:13.839Z] OK: 6 MiB in 15 packages

[2022-01-07T16:03:15.298Z] Removing intermediate container 044ba89f7e21

[2022-01-07T16:03:15.298Z]  ---> cf8e94e15dee

[2022-01-07T16:03:15.298Z] Step 4/37 : RUN apk add --update bash git make build-base

[2022-01-07T16:03:15.298Z]  ---> Running in 16b0ba8cb4e2

[2022-01-07T16:03:15.865Z] (1/30) Installing ncurses-terminfo-base (6.3_p20211120-r0)

[2022-01-07T16:03:16.124Z] (2/30) Installing ncurses-libs (6.3_p20211120-r0)

[2022-01-07T16:03:16.124Z] (3/30) Installing readline (8.1.1-r0)

[2022-01-07T16:03:16.124Z] (4/30) Installing bash (5.1.8-r0)

[2022-01-07T16:03:16.124Z] Executing bash-5.1.8-r0.post-install

[2022-01-07T16:03:16.124Z] (5/30) Installing libgcc (10.3.1_git20211027-r0)

[2022-01-07T16:03:16.124Z] (6/30) Installing libstdc++ (10.3.1_git20211027-r0)

[2022-01-07T16:03:16.124Z] (7/30) Installing binutils (2.37-r3)

[2022-01-07T16:03:16.124Z] (8/30) Installing libmagic (5.41-r0)

[2022-01-07T16:03:16.124Z] (9/30) Installing file (5.41-r0)

[2022-01-07T16:03:16.124Z] (10/30) Installing libgomp (10.3.1_git20211027-r0)

[2022-01-07T16:03:16.124Z] (11/30) Installing libatomic (10.3.1_git20211027-r0)

[2022-01-07T16:03:16.124Z] (12/30) Installing libgphobos (10.3.1_git20211027-r0)

[2022-01-07T16:03:16.383Z] (13/30) Installing gmp (6.2.1-r0)

[2022-01-07T16:03:16.383Z] (14/30) Installing isl22 (0.22-r0)

[2022-01-07T16:03:16.383Z] (15/30) Installing mpfr4 (4.1.0-r0)

[2022-01-07T16:03:16.383Z] (16/30) Installing mpc1 (1.2.1-r0)

[2022-01-07T16:03:16.383Z] (17/30) Installing gcc (10.3.1_git20211027-r0)

[2022-01-07T16:03:17.331Z] (18/30) Installing musl-dev (1.2.2-r7)

[2022-01-07T16:03:17.332Z] (19/30) Installing libc-dev (0.7.2-r3)

[2022-01-07T16:03:17.332Z] (20/30) Installing g++ (10.3.1_git20211027-r0)

[2022-01-07T16:03:17.590Z] (21/30) Installing make (4.3-r0)

[2022-01-07T16:03:17.590Z] (22/30) Installing fortify-headers (1.1-r1)

[2022-01-07T16:03:17.590Z] (23/30) Installing patch (2.7.6-r7)

[2022-01-07T16:03:17.590Z] (24/30) Installing build-base (0.5-r2)

[2022-01-07T16:03:17.590Z] (25/30) Installing brotli-libs (1.0.9-r5)

[2022-01-07T16:03:17.590Z] (26/30) Installing nghttp2-libs (1.46.0-r0)

[2022-01-07T16:03:17.590Z] (27/30) Installing libcurl (7.80.0-r0)

[2022-01-07T16:03:17.590Z] (28/30) Installing expat (2.4.1-r0)

[2022-01-07T16:03:17.590Z] (29/30) Installing pcre2 (10.39-r0)

[2022-01-07T16:03:17.590Z] (30/30) Installing git (2.34.1-r0)

[2022-01-07T16:03:17.849Z] Executing busybox-1.34.1-r3.trigger

[2022-01-07T16:03:17.849Z] OK: 212 MiB in 45 packages

[2022-01-07T16:03:26.207Z] Removing intermediate container 16b0ba8cb4e2

[2022-01-07T16:03:26.207Z]  ---> 9220c3b38f1a

[2022-01-07T16:03:26.207Z] Step 5/37 : COPY --from=golang:1.17-alpine /usr/local/go/ /usr/local/go/

[2022-01-07T16:03:26.465Z] 1.17-alpine: Pulling from library/golang

[2022-01-07T16:03:26.465Z] 59bf1c3509f3: Already exists

[2022-01-07T16:03:26.465Z] 666ba61612fd: Pulling fs layer

[2022-01-07T16:03:26.465Z] 8ed8ca486205: Pulling fs layer

[2022-01-07T16:03:26.465Z] 4fe6c5a15a65: Pulling fs layer

[2022-01-07T16:03:26.465Z] 22a0e8bec74d: Pulling fs layer

[2022-01-07T16:03:26.465Z] 22a0e8bec74d: Waiting

[2022-01-07T16:03:26.757Z] 8ed8ca486205: Verifying Checksum

[2022-01-07T16:03:26.757Z] 8ed8ca486205: Download complete

[2022-01-07T16:03:26.757Z] 666ba61612fd: Download complete

[2022-01-07T16:03:27.059Z] 22a0e8bec74d: Verifying Checksum

[2022-01-07T16:03:27.059Z] 22a0e8bec74d: Download complete

[2022-01-07T16:03:27.361Z] 666ba61612fd: Pull complete

[2022-01-07T16:03:27.619Z] 8ed8ca486205: Pull complete

[2022-01-07T16:03:29.058Z] 4fe6c5a15a65: Verifying Checksum

[2022-01-07T16:03:29.058Z] 4fe6c5a15a65: Download complete

[2022-01-07T16:03:47.466Z] 4fe6c5a15a65: Pull complete

[2022-01-07T16:03:47.466Z] 22a0e8bec74d: Pull complete

[2022-01-07T16:03:47.466Z] Digest: sha256:519c827ec22e5cf7417c9ff063ec840a446cdd30681700a16cf42eb43823e27c

[2022-01-07T16:03:47.466Z] Status: Downloaded newer image for golang:1.17-alpine

[2022-01-07T16:04:14.226Z]  ---> 382d40c5acfb

[2022-01-07T16:04:14.226Z] Step 6/37 : ENV PATH="/usr/local/go/bin:${PATH}"

[2022-01-07T16:04:14.226Z]  ---> Running in 9d577d293fc2

[2022-01-07T16:04:14.226Z] Removing intermediate container 9d577d293fc2

[2022-01-07T16:04:14.226Z]  ---> 2fe58dc98c0d

[2022-01-07T16:04:14.226Z] Step 7/37 : WORKDIR /workspace

[2022-01-07T16:04:14.226Z]  ---> Running in f437ff8d6249

[2022-01-07T16:04:14.226Z] Removing intermediate container f437ff8d6249

[2022-01-07T16:04:14.226Z]  ---> 9e881507e324

[2022-01-07T16:04:14.226Z] Step 8/37 : COPY cmd  /workspace/cmd

[2022-01-07T16:04:14.226Z]  ---> 084ffc7efac6

[2022-01-07T16:04:14.226Z] Step 9/37 : COPY config.yaml /workspace/config.yaml

[2022-01-07T16:04:14.226Z]  ---> fa71e3d71a3e

[2022-01-07T16:04:14.226Z] Step 10/37 : COPY pkg /workspace/pkg

[2022-01-07T16:04:14.226Z]  ---> c6759b039274

[2022-01-07T16:04:14.227Z] Step 11/37 : COPY tools /workspace/tools

[2022-01-07T16:04:14.227Z]  ---> 18e14afe8fdc

[2022-01-07T16:04:14.227Z] Step 12/37 : COPY VERSION /workspace/VERSION

[2022-01-07T16:04:14.227Z]  ---> eda2165d9e75

[2022-01-07T16:04:14.227Z] Step 13/37 : COPY .git /workspace/.git

[2022-01-07T16:04:14.227Z]  ---> 072b6921b17b

[2022-01-07T16:04:14.227Z] Step 14/37 : COPY Makefile /workspace/Makefile

[2022-01-07T16:04:14.227Z]  ---> 4405ede2a5d7

[2022-01-07T16:04:14.227Z] Step 15/37 : COPY config /workspace/config

[2022-01-07T16:04:14.227Z]  ---> 3d3a4506a542

[2022-01-07T16:04:14.227Z] Step 16/37 : COPY go.* build/linux/aoc_linux_x86_6* /workspace/

[2022-01-07T16:04:14.561Z]  ---> 59a2fdeb8901

[2022-01-07T16:04:14.561Z] Step 17/37 : RUN if [[ -f /workspace/aoc_linux_x86_64 ]] ;     then         echo Copying binary         && mkdir -p /workspace/build/linux         && cp /workspace/aoc_linux_x86_64 /workspace/build/linux/aoc_linux_x86_64 ;     else         echo Building binary         && go mod download          && make amd64-build ;     fi

[2022-01-07T16:04:14.561Z]  ---> Running in 203e309a8923

[2022-01-07T16:04:15.161Z] Building binary

[2022-01-07T17:07:00.610Z] make: cat: Operation not permitted

[2022-01-07T17:07:00.610Z] make: /bin/sh: Operation not permitted

[2022-01-07T17:07:00.610Z] make: /bin/sh: Operation not permitted

[2022-01-07T17:07:00.610Z] make: go: Operation not permitted

[2022-01-07T17:07:00.610Z] make: /bin/sh: Operation not permitted

[2022-01-07T17:07:00.610Z] cd /workspace/tools/linters && GOBIN=/workspace/bin go install go.opentelemetry.io/build-tools/multimod

[2022-01-07T17:07:00.610Z] make: /bin/sh: Operation not permitted

[2022-01-07T17:07:00.610Z] make: *** [Makefile:182: install-tools] Error 127

[2022-01-07T17:07:00.610Z] The command '/bin/sh -c if [[ -f /workspace/aoc_linux_x86_64 ]] ;     then         echo Copying binary         && mkdir -p /workspace/build/linux         && cp /workspace/aoc_linux_x86_64 /workspace/build/linux/aoc_linux_x86_64 ;     else         echo Building binary         && go mod download          && make amd64-build ;     fi' returned a non-zero code: 2