Macos signing in cli release build broken #1993
Comments
This was referenced Aug 26, 2024
ddelnano
added a commit
that referenced
this issue
Sep 4, 2024
… to parse (#2005) Summary: Fix bug where cli manifests with `AT_DARWIN_ARM64` ArtifactTypes fail to parse This was an error I ran into when testing #1997. You can see the failure output from the [following build](https://github.com/pixie-io/pixie/actions/runs/10705210930/job/29681122220?pr=1997) (also copied below). ``` INFO: Running command line: bazel-bin/src/utils/artifacts/manifest_updater/manifest_updater_/manifest_updater '--manifest_updates=/__w/pixie/pixie/manifest_updates.json' '--manifest_path=/__w/pixie/pixie/gh-pages/artifacts/manifest.json' time="2024-09-04T16:17:53Z" level=fatal msg="failed to read manifest updates" error="unknown value \"AT_DARWIN_ARM64\" for enum px.versions.ArtifactType" ``` Relevant Issues: #1993 Type of change: /kind bugfix Test Plan: Built the manifest_updater binary and verified it no longer fails with a parse error ``` $ ./manifest_updater --manifest_updates manifest_updates.json --manifest_path=manifest.json $ ``` --------- Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
ddelnano
added a commit
that referenced
this issue
Sep 5, 2024
Summary: Use maintained gon fork to address Apple signing deprecation This is a continuation of #1994. Rather than using the macos tools directly, we can continue using gon by switching to the maintained fork ([github.com/Bearer/gon](https://github.com/Bearer/gon)) Relevant Issues: Closes #1993 Type of change: /kind bugfix Test Plan: cli-release GitHub workflow [succeeds](https://github.com/pixie-io/pixie/actions/runs/10724342153/job/29740236170) when built from this branch Changelog Message: Fix macos signing for px cli releases --------- Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
Merged
1 task
ddelnano
added a commit
that referenced
this issue
Sep 6, 2024
… by podman maintainers (#2008) Summary: Fix production cli release build by using seccomp profile recommended by podman maintainers The cli pre-release builds do not trigger our RPM and deb packaging. This packaging step is accomplished by running podman containers and failed during our most recent production cli release (error below): ``` + podman run -i --rm -v /tmp/tmp.xkTyDurlVW:/src/ -v /__w/pixie/pixie:/image docker.io/cdrx/fpm-fedora:24 fpm -f -p /image/pixie-px-0.8.3.x86_64.rpm -s dir -t rpm -n pixie-px -v 0.8.3 --prefix /usr/local/bin px time="2024-09-05T22:47:26Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers" cannot clone: Operation not permitted Error: cannot re-exec process ``` This is an attempt to remediate the issue by following what was recommended by podman's maintainers in a [past issue](containers/podman#10802 (comment)). Relevant Issues: #1993 Type of change: /kind bugfix Test Plan: Requires building a production release cli since this logic is skipped otherwise - [x] Verified this seccomp profile is a dependency of the podman Ubuntu package and installed in our dev image that the CI job uses. ``` ddelnano@dev-vm:~/code/pixie-worktree (ddelnano/address-unable-to-clone-error) $ ./scripts/run_docker.sh pwd && ls -alh /usr/share/containers/seccomp.json sha256:e8d76daa1fe01efdff68cacf982f00a94a674b1d450f8eab11b98f5bd6a5c397 grep: /etc/bazelrc: No such file or directory Run Parameters: Build Buddy: Disabled Shell: /bin/bash + docker run --rm --hostname px-dev-docker-dev-vm.us-west1-a.c.endless-datum-422018.internal -it -v /dev/shm:/dev/shm -v /home/ddelnano:/home/ddelnano -v /var/run/docker.sock:/var/run/docker.sock -v /home/ddelnano/code/pixie-worktree:/px/src/px.dev/pixie --network=host -v /usr/local/bin/px:/bin/px px_dev_image:202405102250 /bin/bash -c pwd /px/src/px.dev/pixie -rw-r--r-- 1 root root 17K Feb 1 2024 /usr/share/containers/seccomp.json ddelnano@px-dev-docker-dev-vm:/px/src/px.dev/pixie (ddelnano/address-unable-to-clone-error) $ dpkg -S /usr/share/containers/seccomp.json golang-github-containers-common: /usr/share/containers/seccomp.json ddelnano@px-dev-docker-dev-vm:/px/src/px.dev/pixie (ddelnano/address-unable-to-clone-error) $ apt-cache rdepends golang-github-containers-common | grep podman podman ``` Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
ddelnano
added a commit
that referenced
this issue
Sep 6, 2024
…ers (#2010) Summary: Another attempt to remediate user namespace issues with ubuntu 24.04 runners This reverts the failed previous attempt and re-enables user namespaces on ubuntu 24.04. This is a well known behavior change between Ubuntu 22.04 and 24.04 (actions/runner-images#10443 (comment)). Since podman is running rootless, I think it's possible that the user namespace creation (clone syscall) is failing. Relevant Issues: #1993 Type of change: /kind bugfix Test Plan: Run another production release build after merging this - Verified that this sysctl cli invocation is similar to our [existing example](https://github.com/pixie-io/pixie/blob/aa1f72a69f0a072d97accd721081f89cb8108fd1/.github/workflows/build_and_test.yaml#L87) --------- Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
aimichelle
pushed a commit
that referenced
this issue
Sep 6, 2024
Summary: Use privileged container for running cli build release steps This reverts my second attempt at fixing the issue with our recent cli builds and tries to address the issue by running the GitHub action's container privileged. This is to address the following error seen in recent cli release builds: ``` + podman run -i --rm -v /tmp/tmp.xkTyDurlVW:/src/ -v /__w/pixie/pixie:/image docker.io/cdrx/fpm-fedora:24 fpm -f -p /image/pixie-px-0.8.3.x86_64.rpm -s dir -t rpm -n pixie-px -v 0.8.3 --prefix /usr/local/bin px time="2024-09-05T22:47:26Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers" cannot clone: Operation not permitted Error: cannot re-exec process ``` Relevant Issues: #1993 Type of change: /kind bugfix Test Plan: Reproduced the `clone: Operation not permitted` issue on my dev VM. Verified that adding `--privileged` no longer sees the issue ``` # Note: running podman without sudo runs into an issue before the clone issue happens. # This may not be the most representative test but hopefully it reproduces the issue closely enough $ ./scripts/run_docker.sh sha256:e8d76daa1fe01efdff68cacf982f00a94a674b1d450f8eab11b98f5bd6a5c397 grep: /etc/bazelrc: No such file or directory Run Parameters: Build Buddy: Disabled Shell: /bin/bash + docker run --rm --hostname px-dev-docker-dev-vm.us-west1-a.c.endless-datum-422018.internal -it -v /dev/shm:/dev/shm -v /home/ddelnano:/home/ddelnano -v /var/run/docker.sock:/var/run/docker.sock -v /home/ddelnano/code/pixie-worktree:/px/src/px.dev/pixie --network=host -v /usr/local/bin/px:/bin/px px_dev_image:202405102250 /bin/bash $ sudo podman info sudo: unable to resolve host px-dev-docker-dev-vm.us-west1-a.c.endless-datum-422018.internal: Name or service not known WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers cannot clone: Operation not permitted Error: cannot re-exec process # Run same ./scripts/run_docker.sh but with --privileged added $ docker run --rm --hostname px-dev-docker-dev-vm.us-west1-a.c.endless-datum-422018.internal -it -v /dev/shm:/dev/shm -v /home/ddelnano:/home/ddelnano -v /var/run/docker.sock:/var/run/docker.sock -v /home/ddelnano/code/pixie-worktree:/px/src/px.dev/pixie --network=host -v /usr/local/bin/px:/bin/px --privileged px_dev_image:202405102250 /bin/bash # podman info succeeds $ sudo podman info sudo: unable to resolve host px-dev-docker-dev-vm.us-west1-a.c.endless-datum-422018.internal: Name or service not known host: arch: amd64 ``` --------- Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
1 task
ddelnano
added a commit
that referenced
this issue
Sep 10, 2024
…tions (#2012) Summary: Add fpm to dev container image to avoid podman in docker in GitHub actions The production cli release build fails on these steps. The container running the job is unable to create new namespaces and runs into the following error: ``` + podman run -i --rm -v /tmp/tmp.oCPtyljlYd:/src/ -v /__w/pixie/pixie:/image docker.io/cdrx/fpm-fedora:24 fpm -f -p /image/pixie-px-0.8.3.x86_64.rpm -s dir -t rpm -n pixie-px -v 0.8.3 --prefix /usr/local/bin px time="2024-09-06T18:10:15Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers" cannot clone: Operation not permitted Error: cannot re-exec process ``` Rather than fix the namespace creation problem, which has been difficult to debug, this installs the necessary tools within the container. This will allow us to avoid using podman to run `fpm`. This change also reverts the previous attempt at fixing this and removes genny from the container image. See [this comment](#2012 (comment)) for more details on the genny removal. Relevant Issues: #1993 Type of change: /kind bugfix Test Plan: Verified that building the deb and rpm packages works from the ubuntu based dev container image and that installing the packages is successful in their respective distros ``` $ docker run --entrypoint bash -it docker.io/ddelnano/dev_image:202409082304 root@f65d4d86e9be:/src# fpm -f -s dir -t rpm -n pixie-px -v 0.8.3 --prefix /usr/local/bin -p pixie-px-0.8.3.x86_64.rpm px Created package {:path=>"pixie-px-0.8.3.x86_64.rpm"} root@f65d4d86e9be:/src# fpm -f -s dir -t deb -n pixie-px -v 0.8.3 --prefix /usr/local/bin -p pixie-px-0.8.3.deb px Created package {:path=>"pixie-px-0.8.3.deb"} # Install the rpm package and test px in a fedora container $ docker run --entrypoint bash -it -v $(pwd):/src/ -w /src fedora [root@90a53608bae0 src]# rpm -i pixie-px-0.8.3.x86_64.rpm [root@90a53608bae0 src]# px --version Pixie CLI ``` - Verified chef works against a Ubuntu noble VM - [ ] Ran `#ci:ignore-deps` and `#ci:bpf-build-all-kernels` build due to genny removal --------- Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
ddelnano
added a commit
that referenced
this issue
Sep 10, 2024
#2015) Summary: Use fpm binary from within dev_image_extras container for cli releases With #2012 merged, the `dev_image_with_extras` container image now contains `fpm`. This change removes the use of podman to avoid the permission issues seen in recent cli builds. This should allow the production cli release build to succeed. Relevant Issues: Closes #1993 Type of change: /kind bugfix Test Plan: Verified that `fpm` exists in most recent image and packaging building commands work (see testing on #2012). Running a production cli release build is needed after this is merged to complete the testing. ``` $ ./scripts/run_docker.sh sha256:c14b06e5753d4d9c6b2ff8671843043113cdcfa3f3cb36a492a665d71dc42fad grep: /etc/bazelrc: No such file or directory Run Parameters: Build Buddy: Disabled Shell: /bin/bash + docker run --rm --hostname px-dev-docker-dev-vm.us-west1-a.c.endless-datum-422018.internal -it -v /dev/shm:/dev/shm -v /home/ddelnano:/home/ddelnano -v /var/run/docker.sock:/var/run/docker.sock -v /home/ddelnano/code/pixie-worktree:/px/src/px.dev/pixie --network=host -v /usr/local/bin/px:/bin/px px_dev_image:202409092344 /bin/bash ddelnano@px-dev-docker-dev-vm:/px/src/px.dev/pixie (ddelnano/use-fpm-within-dev_image_extras-container) $ fpm --version 1.15.1 ``` Signed-off-by: Dom Del Nano <ddelnano@gmail.com>
ddelnano
added a commit
to ddelnano/pixie
that referenced
this issue
Sep 23, 2024
… to parse (pixie-io#2005) Summary: Fix bug where cli manifests with `AT_DARWIN_ARM64` ArtifactTypes fail to parse This was an error I ran into when testing pixie-io#1997. You can see the failure output from the [following build](https://github.com/pixie-io/pixie/actions/runs/10705210930/job/29681122220?pr=1997) (also copied below). ``` INFO: Running command line: bazel-bin/src/utils/artifacts/manifest_updater/manifest_updater_/manifest_updater '--manifest_updates=/__w/pixie/pixie/manifest_updates.json' '--manifest_path=/__w/pixie/pixie/gh-pages/artifacts/manifest.json' time="2024-09-04T16:17:53Z" level=fatal msg="failed to read manifest updates" error="unknown value \"AT_DARWIN_ARM64\" for enum px.versions.ArtifactType" ``` Relevant Issues: pixie-io#1993 Type of change: /kind bugfix Test Plan: Built the manifest_updater binary and verified it no longer fails with a parse error ``` $ ./manifest_updater --manifest_updates manifest_updates.json --manifest_path=manifest.json $ ``` --------- Signed-off-by: Dom Del Nano <ddelnano@gmail.com> GitOrigin-RevId: afffb8e
ddelnano
added a commit
to ddelnano/pixie
that referenced
this issue
Sep 23, 2024
… by podman maintainers (pixie-io#2008) Summary: Fix production cli release build by using seccomp profile recommended by podman maintainers The cli pre-release builds do not trigger our RPM and deb packaging. This packaging step is accomplished by running podman containers and failed during our most recent production cli release (error below): ``` + podman run -i --rm -v /tmp/tmp.xkTyDurlVW:/src/ -v /__w/pixie/pixie:/image docker.io/cdrx/fpm-fedora:24 fpm -f -p /image/pixie-px-0.8.3.x86_64.rpm -s dir -t rpm -n pixie-px -v 0.8.3 --prefix /usr/local/bin px time="2024-09-05T22:47:26Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers" cannot clone: Operation not permitted Error: cannot re-exec process ``` This is an attempt to remediate the issue by following what was recommended by podman's maintainers in a [past issue](containers/podman#10802 (comment)). Relevant Issues: pixie-io#1993 Type of change: /kind bugfix Test Plan: Requires building a production release cli since this logic is skipped otherwise - [x] Verified this seccomp profile is a dependency of the podman Ubuntu package and installed in our dev image that the CI job uses. ``` ddelnano@dev-vm:~/code/pixie-worktree (ddelnano/address-unable-to-clone-error) $ ./scripts/run_docker.sh pwd && ls -alh /usr/share/containers/seccomp.json sha256:e8d76daa1fe01efdff68cacf982f00a94a674b1d450f8eab11b98f5bd6a5c397 grep: /etc/bazelrc: No such file or directory Run Parameters: Build Buddy: Disabled Shell: /bin/bash + docker run --rm --hostname px-dev-docker-dev-vm.us-west1-a.c.endless-datum-422018.internal -it -v /dev/shm:/dev/shm -v /home/ddelnano:/home/ddelnano -v /var/run/docker.sock:/var/run/docker.sock -v /home/ddelnano/code/pixie-worktree:/px/src/px.dev/pixie --network=host -v /usr/local/bin/px:/bin/px px_dev_image:202405102250 /bin/bash -c pwd /px/src/px.dev/pixie -rw-r--r-- 1 root root 17K Feb 1 2024 /usr/share/containers/seccomp.json ddelnano@px-dev-docker-dev-vm:/px/src/px.dev/pixie (ddelnano/address-unable-to-clone-error) $ dpkg -S /usr/share/containers/seccomp.json golang-github-containers-common: /usr/share/containers/seccomp.json ddelnano@px-dev-docker-dev-vm:/px/src/px.dev/pixie (ddelnano/address-unable-to-clone-error) $ apt-cache rdepends golang-github-containers-common | grep podman podman ``` Signed-off-by: Dom Del Nano <ddelnano@gmail.com> GitOrigin-RevId: aa1f72a
ddelnano
added a commit
to ddelnano/pixie
that referenced
this issue
Sep 23, 2024
…ers (pixie-io#2010) Summary: Another attempt to remediate user namespace issues with ubuntu 24.04 runners This reverts the failed previous attempt and re-enables user namespaces on ubuntu 24.04. This is a well known behavior change between Ubuntu 22.04 and 24.04 (actions/runner-images#10443 (comment)). Since podman is running rootless, I think it's possible that the user namespace creation (clone syscall) is failing. Relevant Issues: pixie-io#1993 Type of change: /kind bugfix Test Plan: Run another production release build after merging this - Verified that this sysctl cli invocation is similar to our [existing example](https://github.com/pixie-io/pixie/blob/aa1f72a69f0a072d97accd721081f89cb8108fd1/.github/workflows/build_and_test.yaml#L87) --------- Signed-off-by: Dom Del Nano <ddelnano@gmail.com> GitOrigin-RevId: 5dc122a
ddelnano
added a commit
to ddelnano/pixie
that referenced
this issue
Sep 23, 2024
…tions (pixie-io#2012) Summary: Add fpm to dev container image to avoid podman in docker in GitHub actions The production cli release build fails on these steps. The container running the job is unable to create new namespaces and runs into the following error: ``` + podman run -i --rm -v /tmp/tmp.oCPtyljlYd:/src/ -v /__w/pixie/pixie:/image docker.io/cdrx/fpm-fedora:24 fpm -f -p /image/pixie-px-0.8.3.x86_64.rpm -s dir -t rpm -n pixie-px -v 0.8.3 --prefix /usr/local/bin px time="2024-09-06T18:10:15Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers" cannot clone: Operation not permitted Error: cannot re-exec process ``` Rather than fix the namespace creation problem, which has been difficult to debug, this installs the necessary tools within the container. This will allow us to avoid using podman to run `fpm`. This change also reverts the previous attempt at fixing this and removes genny from the container image. See [this comment](pixie-io#2012 (comment)) for more details on the genny removal. Relevant Issues: pixie-io#1993 Type of change: /kind bugfix Test Plan: Verified that building the deb and rpm packages works from the ubuntu based dev container image and that installing the packages is successful in their respective distros ``` $ docker run --entrypoint bash -it docker.io/ddelnano/dev_image:202409082304 root@f65d4d86e9be:/src# fpm -f -s dir -t rpm -n pixie-px -v 0.8.3 --prefix /usr/local/bin -p pixie-px-0.8.3.x86_64.rpm px Created package {:path=>"pixie-px-0.8.3.x86_64.rpm"} root@f65d4d86e9be:/src# fpm -f -s dir -t deb -n pixie-px -v 0.8.3 --prefix /usr/local/bin -p pixie-px-0.8.3.deb px Created package {:path=>"pixie-px-0.8.3.deb"} # Install the rpm package and test px in a fedora container $ docker run --entrypoint bash -it -v $(pwd):/src/ -w /src fedora [root@90a53608bae0 src]# rpm -i pixie-px-0.8.3.x86_64.rpm [root@90a53608bae0 src]# px --version Pixie CLI ``` - Verified chef works against a Ubuntu noble VM - [ ] Ran `#ci:ignore-deps` and `#ci:bpf-build-all-kernels` build due to genny removal --------- Signed-off-by: Dom Del Nano <ddelnano@gmail.com> GitOrigin-RevId: 07fff00
ddelnano
added a commit
to ddelnano/pixie
that referenced
this issue
Sep 23, 2024
pixie-io#2015) Summary: Use fpm binary from within dev_image_extras container for cli releases With pixie-io#2012 merged, the `dev_image_with_extras` container image now contains `fpm`. This change removes the use of podman to avoid the permission issues seen in recent cli builds. This should allow the production cli release build to succeed. Relevant Issues: Closes pixie-io#1993 Type of change: /kind bugfix Test Plan: Verified that `fpm` exists in most recent image and packaging building commands work (see testing on pixie-io#2012). Running a production cli release build is needed after this is merged to complete the testing. ``` $ ./scripts/run_docker.sh sha256:c14b06e5753d4d9c6b2ff8671843043113cdcfa3f3cb36a492a665d71dc42fad grep: /etc/bazelrc: No such file or directory Run Parameters: Build Buddy: Disabled Shell: /bin/bash + docker run --rm --hostname px-dev-docker-dev-vm.us-west1-a.c.endless-datum-422018.internal -it -v /dev/shm:/dev/shm -v /home/ddelnano:/home/ddelnano -v /var/run/docker.sock:/var/run/docker.sock -v /home/ddelnano/code/pixie-worktree:/px/src/px.dev/pixie --network=host -v /usr/local/bin/px:/bin/px px_dev_image:202409092344 /bin/bash ddelnano@px-dev-docker-dev-vm:/px/src/px.dev/pixie (ddelnano/use-fpm-within-dev_image_extras-container) $ fpm --version 1.15.1 ``` Signed-off-by: Dom Del Nano <ddelnano@gmail.com> GitOrigin-RevId: 4528883
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When trying to release the cli this week, the build failed with the following error:
The gon tool is no longer supported, so we should migrate to
notarytoolwhich is the currently recommended tool by Apple.The text was updated successfully, but these errors were encountered: