Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refactor Beat packaging and cross-building
This refactors Beat packaging to use a declarative YAML based packaging
specification. This makes it easier to customize the contents of packages (e.g.
adding additional X-Pack content or simply tailoring what's included for single
Beat).
The specification itself is pretty simple. It consists of package metadata and a
list of files to include. The values can be templated which allows for a single
package spec to be reused across Beats. Here's an example spec for an OSS
Windows zip package.
```
spec:
name: '{{.BeatName}}-oss'
service_name: '{{.BeatServiceName}}'
os: '{{.GOOS}}'
arch: '{{.PackageArch}}'
vendor: '{{.BeatVendor}}'
version: '{{ beat_version }}'
license: ASL 2.0
url: '{{.BeatURL}}'
description: '{{.BeatDescription}}'
files:
'{{.BeatName}}{{.BinaryExt}}':
source: build/golang-crossbuild/{{.BeatName}}-{{.GOOS}}-{{.Platform.Arch}}{{.BinaryExt}}
mode: 0755
fields.yml:
source: fields.yml
mode: 0644
LICENSE.txt:
source: '{{ repo.RootDir }}/licenses/APACHE-LICENSE-2.0.txt'
mode: 0644
NOTICE.txt:
source: '{{ repo.RootDir }}/NOTICE.txt'
mode: 0644
README.md:
template: '{{ elastic_beats_dir }}/dev-tools/mage/templates/common/README.md.tmpl'
mode: 0644
.build_hash.txt:
content: >
{{ commit }}
mode: 0644
'{{.BeatName}}.reference.yml':
source: '{{.BeatName}}.reference.yml'
mode: 0644
'{{.BeatName}}.yml':
source: '{{.BeatName}}.yml'
mode: 0600
config: true
kibana:
source: _meta/kibana.generated
mode: 0644
install-service-{{.BeatName}}.ps1:
template: '{{ elastic_beats_dir }}/dev-tools/mage/templates/windows/install-service.ps1.tmpl'
mode: 0755
uninstall-service-{{.BeatName}}.ps1:
template: '{{ elastic_beats_dir }}/dev-tools/mage/templates/windows/uninstall-service.ps1.tmpl'
mode: 0755
```
Each Beat has two build targets for packaging.
- `make snapshot`
- `make release`
The set of target platforms can be influenced by the `PLATFORMS` environment
variable which accepts an platform selection expression. For example to add
ARMv7 (for your Raspberry Pi) to the default set of platforms you would use
`PLATFORMS='+linux/armv7' make snapshot`. Or to only build for Windows set
`PLATFORMS='windows'`. Full details can be found in the godocs for
`NewPlatformList`.
For the release manager there are two new top-level targets that take care of
ensuring that the proper Go version is used. The naming here aligns with what
several of the other projects are using for their release manager targets.
- `make release-manager-snapshot`
- `make release-manager-release`
Build Process Details
----
Below is the command line output of building and packaging Packetbeat for
linux/armv7 only. I'll describe each step in the build process.
```
$ PLATFORMS='+all linux/armv7' make snapshot
Installing mage from vendor
>> golangCrossBuild: Building for linux/armv7
>> buildGoDaemon: Building for linux/armv7
>> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=arm-linux-gnueabihf-gcc, CXX=arm-linux-gnueabihf-g++, GOARCH=arm, GOARM=7, GOOS=linux, PLATFORM_ID=linux-armv7]
>> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=arm-linux-gnueabihf-gcc, CXX=arm-linux-gnueabihf-g++, GOARCH=arm, GOARM=7, GOOS=linux, PLATFORM_ID=linux-armv7]
grammar.y: warning: 38 shift/reduce conflicts [-Wconflicts-sr]
>> package: Building packetbeat type=tar.gz for platform=linux/armv7
>> package: Building packetbeat-oss type=deb for platform=linux/armv7
>> package: Building packetbeat type=rpm for platform=linux/armv7
>> package: Building packetbeat type=deb for platform=linux/armv7
>> package: Building packetbeat-oss type=tar.gz for platform=linux/armv7
>> package: Building packetbeat-oss type=rpm for platform=linux/armv7
>> Testing package contents
package ran for 1m6.597416206s
$ tree build/distributions/
build/distributions/
├── packetbeat-oss-7.0.0-alpha1-SNAPSHOT-armhf.deb
├── packetbeat-oss-7.0.0-alpha1-SNAPSHOT-armhf.deb.sha512
├── packetbeat-oss-7.0.0-alpha1-SNAPSHOT-armhfp.rpm
├── packetbeat-oss-7.0.0-alpha1-SNAPSHOT-armhfp.rpm.sha512
├── packetbeat-oss-7.0.0-alpha1-SNAPSHOT-linux-armv7.tar.gz
└── packetbeat-oss-7.0.0-alpha1-SNAPSHOT-linux-armv7.tar.gz.sha512
```
1. Install [mage](https://magefile.org/) from the vendor directory to `$GOPATH/bin`.
Mage is used to provide a simple gmake like wrapper around build logic that's
written in Go. Additionally it makes is possible to build and package without
gmake by invoking mage directly [think Windows users]. For example:
```
$ mage -l
Targets:
build builds the Beat binary.
buildGoDaemon builds the go-daemon binary (use crossBuildGoDaemon).
clean cleans all generated files and build artifacts.
crossBuild cross-builds the beat for all target platforms.
crossBuildGoDaemon cross-builds the go-daemon binary using Docker.
golangCrossBuild build the Beat binary inside of the golang-builder.
package packages the Beat for distribution.
testPackages tests the generated packages (i.e.
update updates the generated files (aka make update).
```
2. Cross-build Packetbeat for linux/armv7. Cross-building requires Docker and
uses images hosted at `docker.elastic.co/beats-dev/golang-crossbuild`. The
repo for these images is https://github.com/elastic/golang-crossbuild. These
images give us wider platform support for cross-building than we had.
`mage golangCrossBuild` is invoked inside of the container. This handles
cross-compiling libpcap and then invoking `go build` with the proper args.
3. Cross-build go-daemon for linux/armv7. This is actually done in parallel with
the other cross-builds. GOMAXPROCS determines the number of concurrent jobs.
`mage buildGoDaemon` is invoked inside of the container.
4. After all cross-builds complete, packaging begins. The package types are
decided based on the package specs that are registered for each OS.
Zip and tar.gz files are built natively with Go. RPM and deb packages are
first generated as tar.gz where we have full control over the target file
names, ownership, and modes regardless of the underlying filesystem [think
Windows]. Then FPM is invoked inside of Docker to translate the tar.gz file
into a proper RPM or deb.
5. SHA512 side-car files are generated for each package. Go is used for this so
no special command line tools are needed.
6. The generated packages are inspected with `dev-tools/package_test.go`. This
looks at the contents of the packages to ensure the files have the expected
ownership and modes (e.g. the config file should have 0600).
Changes
- Add Boot2Docker workaround for shared volume permissions
- Mirror the libpcap source to S3
- Use MAX_PARALLEL to control the number of parallel jobs. The default value is
the lesser of the NumCPU and NCPU from the Docker daemon.
- Add jenkins_package.sh for Jenkins.- Loading branch information
1 parent
fe17304
commit 23c4bc0