Build cloud docker images for elastic-agent

[v1v]

What does this PR do?

Build cloud docker images for elastic-agent that requires the metricbeat and filebeat folder in the /opt folder. In addition, to change the entrypoint.

Why is it important?

Provide docker images for the cloud as part of the Beats projects, as part of the cloud initiative.

Cloud dockerfile

FROM docker.elastic.co/beats/elastic-agent:buildversion

COPY build/filebeat /opt/filebeat
COPY build/metricbeat /opt/metricbeat

ENTRYPOINT ["/bin/tini", "--"]
CMD ["/app/apm.sh"]

Test

export PATH=${PATH}:~/work/bin
export GVM_CMD="${HOME}/bin/gvm"
export GO_VERSION=1.16.6
.ci/scripts/install-go.sh
eval "$("${GVM_CMD}" "${GO_VERSION}")"
make
cd x-pack/elastic-agent
PLATFORMS=linux/amd64 mage package

Then, I validated the filebeat and metricbeat folders are stored in the /opt folder:

docker run --rm -ti --entrypoint /bin/ls docker.elastic.co/beats/elastic-agent-cloud:8.0.0 -ltr /opt 
total 8
drwxr-xr-x 5 root root 4096 Sep 29 17:37 filebeat
drwxr-xr-x 5 root root 4096 Sep 29 17:37 metricbeat

docker run --rm -ti --entrypoint /bin/ls docker.elastic.co/beats/elastic-agent-cloud:8.0.0 -ltr /opt/filebeat
total 168096
-rw-r--r--  1 root root     13675 Apr  7 09:40 LICENSE.txt
-rw-r--r--  1 root root   8658333 Sep 20 16:50 NOTICE.txt
drwxr-xr-x 31 root root      4096 Sep 27 14:11 module
drwxr-xr-x  2 root root      4096 Sep 27 14:11 modules.d
drwxr-xr-x  3 root root      4096 Sep 27 14:11 kibana
-rw-r--r--  1 root root   1197883 Sep 27 14:11 fields.yml
-rw-------  1 root root      6899 Sep 27 14:11 metricbeat.yml
-rw-r--r--  1 root root    101197 Sep 27 14:11 metricbeat.reference.yml
-rwxr-xr-x  1 root root 162126224 Sep 27 14:27 metricbeat
-rw-r--r--  1 root root       824 Sep 27 14:28 README.md

docker run --rm -ti --entrypoint /bin/ls docker.elastic.co/beats/elastic-agent:8.0.0 -ltr /opt 
total 0

docker run --rm -ti --entrypoint /bin/ls docker.elastic.co/beats/elastic-agent-ubi8:8.0.0 -ltr /opt 
total 0

docker run --rm -ti --entrypoint /bin/ls docker.elastic.co/beats/elastic-agent-complete:8.0.0 -ltr /opt 
total 0

docker images
REPOSITORY                                                     TAG                                              IMAGE ID       CREATED          SIZE
docker.elastic.co/beats/elastic-agent-complete                 8.0.0                                            2f5a4cccb07b   13 minutes ago   1.56GB
docker.elastic.co/beats/elastic-agent-cloud                    8.0.0                                            9abe05e04115   14 minutes ago   843MB
docker.elastic.co/beats/elastic-agent                          8.0.0                                            6a385ef2124d   14 minutes ago   525MB
docker.elastic.co/beats/elastic-agent-ubi8                     8.0.0                                            4427dd2b7706   14 minutes ago   299MB

Diff

diff --git a/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build/Dockerfile b/build/package/elastic-agent-cloud/elastic-agent-linux-amd64.docker/docker-build/Dockerfile
index a4f0bb94b15..6e0a366e405 100644
--- a/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build/Dockerfile
+++ b/build/package/elastic-agent-cloud/elastic-agent-linux-amd64.docker/docker-build/Dockerfile
@@ -15,13 +15,16 @@ RUN mkdir -p /usr/share/elastic-agent/data /usr/share/elastic-agent/data/elastic
     rm /usr/share/elastic-agent/elastic-agent && \
     ln -s /usr/share/elastic-agent/data/elastic-agent-173757/elastic-agent /usr/share/elastic-agent/elastic-agent && \
     chmod 0755 /usr/share/elastic-agent/data/elastic-agent-*/elastic-agent && \
+    mkdir -p /opt/filebeat /opt/metricbeat && \
+    tar xf /usr/share/elastic-agent/data/elastic-agent-*/downloads/metricbeat-*.tar.gz -C /opt/filebeat --strip-components=1 && \
+    tar xf /usr/share/elastic-agent/data/elastic-agent-*/downloads/filebeat-*.tar.gz -C /opt/metricbeat --strip-components=1 && \
     true
 
 FROM centos:7
 
 # Contains the elastic agent image variant, an empty string for the standard variant
 # or "complete" for the bigger one.
-ENV ELASTIC_AGENT_IMAGE_VARIANT=complete
+ENV ELASTIC_AGENT_IMAGE_VARIANT=
 # Installing jq needs to be installed after epel-release and cannot be in the same yum install command.
 RUN case $(arch) in aarch64) YUM_FLAGS="-x bind-license";; esac; \
     for iter in {1..10}; do \
@@ -33,23 +36,6 @@ RUN case $(arch) in aarch64) YUM_FLAGS="-x bind-license";; esac; \
         exit_code=0 && break || exit_code=$? && echo "yum error: retry $iter in 10s" && sleep 10; \
     done; \
     (exit $exit_code)
-RUN for iter in {1..10}; do \
-        yum -y install atk cups gtk gdk xrandr pango libXcomposite libXcursor libXdamage \
-        libXext libXi libXtst cups-libs libXScrnSaver libXrandr GConf2 \
-        alsa-lib atk gtk3 ipa-gothic-fonts xorg-x11-fonts-100dpi xorg-x11-fonts-75dpi xorg-x11-utils \
-        xorg-x11-fonts-cyrillic xorg-x11-fonts-Type1 xorg-x11-fonts-misc \
-        yum clean all && \
-        exit_code=0 && break || exit_code=$? && echo "yum error: retry $iter in 10s" && sleep 10; \
-    done; \
-    (exit $exit_code)
-ENV NODE_PATH=/usr/share/elastic-agent/.node
-RUN echo \
-    $NODE_PATH \
-    /usr/share/elastic-agent/.config \
-    /usr/share/elastic-agent/.synthetics \
-    /usr/share/elastic-agent/.npm \
-    /usr/share/elastic-agent/.cache \
-    | xargs -IDIR sh -c 'mkdir -p DIR && chmod 0770 DIR'
 
 LABEL \
   org.label-schema.build-date="2021-09-29T17:58:17Z" \
@@ -116,35 +102,13 @@ RUN chmod 0770 /usr/share/elastic-agent
 RUN mkdir /licenses
 COPY --from=home /usr/share/elastic-agent/LICENSE.txt /licenses
 COPY --from=home /usr/share/elastic-agent/NOTICE.txt /licenses
+COPY --from=home /opt /opt
 RUN groupadd --gid 1000 elastic-agent
 RUN useradd -M --uid 1000 --gid 1000 --groups 0 --home /usr/share/elastic-agent elastic-agent
-RUN chown elastic-agent $NODE_PATH
+# Generate folder for a stub command that will be overwritten at runtime
+RUN mkdir /app
+RUN chown elastic-agent /app
 USER elastic-agent
-# Setup synthetics env vars
-ENV ELASTIC_SYNTHETICS_CAPABLE=true
-ENV SUITES_DIR=/usr/share/elastic-agent/suites
-ENV NODE_VERSION=14.17.5
-ENV PATH="$NODE_PATH/node/bin:$PATH"
-# Install the latest version of @elastic/synthetics forcefully ignoring the previously
-# cached node_modules, heartbeat then calls the global executable to run test suites
-# Setup node
-RUN cd /usr/share/elastic-agent/.node \
-  && NODE_DOWNLOAD_URL="" \
-  && case "$(arch)" in \
-       x86_64) \
-           NODE_DOWNLOAD_URL=https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz \
-           ;; \
-       aarch64) \
-           NODE_DOWNLOAD_URL=https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-arm64.tar.xz \
-           ;; \
-       *) \
-           echo >&2 ; echo >&2 "Unsupported architecture \$(arch)" ; echo >&2 ; exit 1 ; \
-           ;; \
-     esac \
-  && mkdir -p node \
-  && curl ${NODE_DOWNLOAD_URL} | tar -xJ --strip 1 -C node \
-  && chmod ug+rwX -R $NODE_PATH \
-  && npm i -g -f @elastic/synthetics && chmod ug+rwX -R $NODE_PATH
 
 # When running under Docker, we must ensure libbeat monitoring pulls cgroup
 # metrics from /sys/fs/cgroup/<subsystem>/, ignoring any paths found in
@@ -152,4 +116,8 @@ RUN cd /usr/share/elastic-agent/.node \
 ENV LIBBEAT_MONITORING_CGROUPS_HIERARCHY_OVERRIDE=/
 
 WORKDIR /usr/share/elastic-agent
-ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/docker-entrypoint"]
+ENTRYPOINT ["/usr/bin/tini", "--"]
+CMD ["/app/apm.sh"]
+# Generate a stub command that will be overwritten at runtime
+RUN echo -e '#!/bin/sh\nexec /usr/local/bin/docker-entrypoint' > /app/apm.sh && \
+    chmod 0555 /app/apm.sh

[v1v]

/packaging

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-10-01T08:02:42.693+0000

• Duration: 234 min 25 sec

• Commit: 2d8119d

Test stats 🧪

Test	Results
Failed	0
Passed	54183
Skipped	5346
Total	59529

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

[v1v]

/test

[v1v]

Elastic-Agent package stage in the Packaging pipeline -> here

[2021-09-27T20:20:54.416Z] + docker push docker.elastic.co/observability-ci/elastic-agent-cloud:pr-28134
[2021-09-27T20:22:17.631Z] + docker push docker.elastic.co/observability-ci/elastic-agent-cloud:pr-28134-amd64
[2021-09-27T20:22:19.657Z] + docker push docker.elastic.co/observability-ci/elastic-agent-cloud:14c5499c4d85a4e05cab3143afe4569b79ce0f71

$ docker pull docker.elastic.co/observability-ci/elastic-agent-cloud:pr-28134
$ docker run --rm -ti --entrypoint /bin/bash docker.elastic.co/observability-ci/elastic-agent-cloud:pr-28134
bash-4.2$ ls -l /opt/
total 273288
-rwxr-xr-x 1 root root 117717984 Sep 27 19:51 filebeat
-rwxr-xr-x 1 root root 162126224 Sep 27 20:04 metricbeat
bash-4.2$ /opt/filebeat 
Exiting: error loading config file: stat filebeat.yml: no such file or directory
bash-4.2$ /opt/metricbeat 
Exiting: error loading config file: stat metricbeat.yml: no such file or directory

[v1v]

/test

[v1v]

/package

[v1v]

@ruflin , IIRC, you mentioned that I could test those cloud docker images locally, any hints how I can do it? At the moment, the e2e pipelines are the ones to validate those changes are not breaking anything in the existing docker images.

[ruflin]

I sent you the docs link via private slack channel as it is in private repo. If you need help on the setup, please ping @michel-laterman , @stuartnelson3 or @simitt as they have done it in the past.

[mdelapenya]

This LGreatTM, although I'd appreciate the feedback from any Beats maintainer, more specifically related to the build system.

[cachedout]

The paths for mb/fb extraction don't look right to me. Could you please double-check?

[v1v]

/package

[mieciu]

A non-binding LGTM after a round of manual testing I've done with @v1v. As I've spoken to the team, I'd like to see:

2021-10-05T12:22:00.360Z	ERROR	cmd/run.go:122	failed to invoke rollback watcher: fork/exec /app/elastic-agent/data/data/elastic-agent-c21249/elastic-agent: no such file or directory

in a separate issue prior to merging this. FYSA @simitt

[blakerouse]

Change looks good to me!

[EricDavisX]

@v1v can we get a link back to any meta or feature ticket this relates to? I'm curious to review specs / AC on what is needed. Are you only intending to include Metricbeat/Filbebeat in the build, and not more? I'd like to see commitment (even if downstream is ok) as to where it is consumed in tests (on Cloud side?) to know the new artifact is covered

[v1v]

@EricDavisX, I just put you in the loop in the internal issue, since it's not public as far as I know.

Are you only intending to include Metricbeat/Filbebeat in the build, and not more?

Nothing else, in the description there is the snippet of the Dockerfile that the cloud image uses.

I'd like to see commitment (even if downstream is ok) as to where it is consumed in tests (on Cloud side?) to know the new artifact is covered

I don't know if the Unified Release (UR) process does some other validations, but in a nutshell, this new docker images, is processed by the UR and copy some tar.gz files. I tried to describe what I did to test this docker image with the needed artifacts in the internal issue. Besides, the cloud mounts the /app volume and some other things, again outside of the scope of this particular issue.

[jlind23]

@v1v did you have time to double check @cachedout comment?

[cachedout]

@v1v did you have time to double check @cachedout comment?

He addressed my concerns. I am just trying to find a bit of time to test this change locally before finishing my review.

[v1v]

@v1v did you have time to double check @cachedout comment?

@jlind23, thanks for asking, I did it in 2d8119d

and @mieciu and I verified that this new docker image ran in a local ECE environment.

[cachedout]

Tested and looks good. Nice job! 🥇