fix: build elastic agent dependencies only on amd64 architecture #26387

kuisathaverat · 2021-06-21T14:43:17Z

What does this PR do?

It does not build Elastic Agent beats dependencies if you are not running on amd64 architecture.

Why is it important?

The current code tries to build amd64 packages in ARM architecture, we only support cross-compile for all architectures on amd64 machines. This causes that the mage package fails when you make it on an ARM host. The change in this PR skip dependencies in other architectures other than amd64.

Checklist

My code follows the style guidelines of this project
[ X I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
~~- [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

Author's Checklist

[ ]

How to test this PR locally

Launch a ubuntu-18.04/amd64 VM
Enter on x-pack/elastic-agent
Run mage package
It will build filebeat, metricbeat, heartbeat, and elastic-agent
Launch a ubuntu-18.04/arm64 VM
Enter on x-pack/elastic-agent
Run mage package
It will build Elastic-Agent only

Related issues

closes Elastic Agent ARM package are failling #26246

Use cases

Screenshots

Logs

elasticmachine · 2021-06-21T14:44:04Z

Pinging @elastic/agent (Team:Agent)

v1v · 2021-06-21T15:05:15Z

/package

kuisathaverat · 2021-06-21T15:52:48Z

There is something more to fix

[2021-06-21T15:35:23.439Z] Error: failed building elastic-agent type=docker for platform=linux/arm64: failed to copy from /heartbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512 to build/package/elastic-agent/elastic-agent-linux-arm64.docker/docker-build/beat/data/elastic-agent-60d82a/downloads/heartbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512: copy failed: cannot stat source file /heartbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512: stat /heartbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz.sha512: no such file or directory

[2021-06-21T15:35:23.440Z] failed building elastic-agent type=docker for platform=linux/arm64: failed to copy from /metricbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz to build/package/elastic-agent-ubi8/elastic-agent-linux-arm64.docker/docker-build/beat/data/elastic-agent-60d82a/downloads/metricbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz: copy failed: cannot stat source file /metricbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz: stat /metricbeat-8.0.0-SNAPSHOT-linux-arm64.tar.gz: no such file or directory

elasticmachine · 2021-06-21T16:03:02Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: kuisathaverat commented: /test
Start Time: 2021-06-22T17:09:10.640+0000
Duration: 128 min 23 sec
Commit: b7db006

Test stats 🧪

Test	Results
Failed	0
Passed	327
Skipped	9
Total	336

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	327
Skipped	9
Total	336

urso · 2021-06-21T16:33:41Z

@blakerouse @michalpristas Can you please have a look?

kuisathaverat · 2021-06-21T17:24:23Z

I follow the issue to PackageSpec.Files but I dunno where this info is loaded for the Docker package, and it is really hard to track.

kuisathaverat · 2021-06-22T14:31:47Z

/test

kuisathaverat · 2021-06-22T15:26:17Z

it feels like progress, now it fails on the Dockerfile


[2021-06-22T15:05:18.079Z] The command '/bin/sh -c mkdir -p /usr/share/elastic-agent/data /usr/share/elastic-agent/data/elastic-agent-4336ce/logs &&     chown -R root:root /usr/share/elastic-agent &&     find /usr/share/elastic-agent -type d -exec chmod 0755 {} \; &&     find /usr/share/elastic-agent -type f -exec chmod 0644 {} \; &&     find /usr/share/elastic-agent/data -type d -exec chmod 0770 {} \; &&     find /usr/share/elastic-agent/data -type f -exec chmod 0660 {} \; &&     rm /usr/share/elastic-agent/elastic-agent &&     ln -s /usr/share/elastic-agent/data/elastic-agent-4336ce/elastic-agent /usr/share/elastic-agent/elastic-agent &&     chmod 0755 /usr/share/elastic-agent/data/elastic-agent-*/elastic-agent &&     true' returned a non-zero code: 1

kuisathaverat · 2021-06-22T17:08:55Z

/test

kuisathaverat · 2021-06-22T17:33:14Z

tested on an ARM machine

ubuntu:~/src/beats/x-pack/elastic-agent$ mage package
>> Building elastic-agent.yml for linux/amd64
>> Building elastic-agent.reference.yml for linux/amd64
>> Building elastic-agent.docker.yml for linux/amd64
>> Buildspec from spec/*.yml to pkg/agent/program/supported.go
>> BuildPGP from GPG-KEY-elasticsearch to pkg/release/pgp.go
>> BuildFleetCfg _meta/elastic-agent.fleet.yml to pkg/agent/application/configuration_embed.go
>> buildGoDaemon: Building for linux/arm64
>> golangCrossBuild: Building for linux/arm64
>> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]
>> Building using: cmd='build/mage-linux-arm64 buildGoDaemon', env=[GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64]
>> package: Building elastic-agent type=docker for platform=linux/arm64
>> package: Building elastic-agent type=docker for platform=linux/arm64
>> Testing package contents
package ran for 1m56.697427224s
ubuntu:~/src/beats/x-pack/elastic-agent$ docker images
REPOSITORY                                      TAG                       IMAGE ID       CREATED          SIZE
docker.elastic.co/beats/elastic-agent-ubi8      8.0.0                     7ffba3832c6d   5 minutes ago    490MB
docker.elastic.co/beats/elastic-agent           8.0.0                     487e486607c6   5 minutes ago    534MB
<none>                                          <none>                    bf204c729369   5 minutes ago    482MB
<none>                                          <none>                    2fa10caab00c   5 minutes ago    482MB
<none>                                          <none>                    495fd7e6f28a   27 minutes ago   490MB
<none>                                          <none>                    078e5d1c7b8e   27 minutes ago   534MB
<none>                                          <none>                    845e45ee7863   27 minutes ago   482MB
<none>                                          <none>                    a66ee03b241f   27 minutes ago   482MB
docker.elastic.co/beats-dev/golang-crossbuild   1.16.5-darwin-debian10    723cbb39d7b0   13 days ago      1.46GB
docker.elastic.co/beats-dev/golang-crossbuild   1.16.5-base-arm-debian9   9ed06fefcf58   13 days ago      795MB
docker.elastic.co/beats-dev/golang-crossbuild   1.16.5-main               e198d41b2bd0   13 days ago      1.76GB
docker.elastic.co/beats-dev/golang-crossbuild   1.16.5-main-debian7       4f883cc5326d   13 days ago      1.08GB
arm64v8/centos                                  7                         e5df02c43685   7 months ago     301MB
registry.access.redhat.com/ubi8/ubi-minimal     8.2                       a8d933bd291e   9 months ago     164MB

kuisathaverat · 2021-06-23T08:33:31Z

The e2e test failed are not related to the change, the same test failed on amd64 and arm64.

kuisathaverat · 2021-06-23T09:52:42Z

I am checking to force to build arm64 binaries on arm64, on #26415

kuisathaverat · 2021-06-23T09:57:39Z

superseded by #26415

fix: build elastic agent dependencies only on amd64 architecture

23d6628

kuisathaverat requested review from ph, urso, michalpristas, blakerouse and kaiyan-sheng June 21, 2021 14:43

kuisathaverat self-assigned this Jun 21, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 21, 2021

kuisathaverat requested a review from a team June 21, 2021 14:43

kuisathaverat added Team:Elastic-Agent Label for the Agent team Team:Automation Label for the Observability productivity team labels Jun 21, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 21, 2021

cachedout approved these changes Jun 21, 2021

View reviewed changes

kuisathaverat added 2 commits June 22, 2021 10:45

fix: update Docker package for ARM

577f997

fix: add commit config

f7c505c

fix: move binary files

b7db006

kuisathaverat mentioned this pull request Jun 22, 2021

x-pack/elastic-agent-packaging-arm-arm - mage package failed #26239

Closed

kuisathaverat closed this Jun 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: build elastic agent dependencies only on amd64 architecture #26387

fix: build elastic agent dependencies only on amd64 architecture #26387

kuisathaverat commented Jun 21, 2021

elasticmachine commented Jun 21, 2021

v1v commented Jun 21, 2021

kuisathaverat commented Jun 21, 2021

elasticmachine commented Jun 21, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

urso commented Jun 21, 2021

kuisathaverat commented Jun 21, 2021

kuisathaverat commented Jun 22, 2021

kuisathaverat commented Jun 22, 2021

kuisathaverat commented Jun 22, 2021

kuisathaverat commented Jun 22, 2021 •

edited

Loading

kuisathaverat commented Jun 23, 2021

kuisathaverat commented Jun 23, 2021

kuisathaverat commented Jun 23, 2021

fix: build elastic agent dependencies only on amd64 architecture #26387

fix: build elastic agent dependencies only on amd64 architecture #26387

Conversation

kuisathaverat commented Jun 21, 2021

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Jun 21, 2021

v1v commented Jun 21, 2021

kuisathaverat commented Jun 21, 2021

elasticmachine commented Jun 21, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

urso commented Jun 21, 2021

kuisathaverat commented Jun 21, 2021

kuisathaverat commented Jun 22, 2021

kuisathaverat commented Jun 22, 2021

kuisathaverat commented Jun 22, 2021

kuisathaverat commented Jun 22, 2021 • edited Loading

kuisathaverat commented Jun 23, 2021

kuisathaverat commented Jun 23, 2021

kuisathaverat commented Jun 23, 2021

elasticmachine commented Jun 21, 2021 •

edited by jenkins-beats-ci bot

Loading

kuisathaverat commented Jun 22, 2021 •

edited

Loading