Expand mage and add build options #73

narph · 2022-07-06T13:24:41Z

Implement build solution using goreleaser (https://goreleaser.com/intro/)

align to stack version
define in .goreleaser.yaml build options

Expand .CI

in progress

Expand mage and add build options

Run mage to list targets

Targets:
  build:all             builds binaries for the all os/arch.
  build:binary          will create the project binaries found in /build/binaries (use `mage build`) ENV PLATFORM = all, darwin, linux, windows, darwin/amd64, darwin/arm64, linux/386, linux/amd64, linux/arm64, windows/386, windows/amd64 ENV SNAPSHOT = true/false
  build:clean           removes the build directory.
  build:testBinaries    checks if the binaries are generated (for now).
  check                 runs all the checks including licence, notice, gomod, git changes
  checkLicense          checks the license headers
  installGoReleaser     target installs goreleaser
  license               should generate the license headers
  notice                generates a NOTICE.txt file for the module.
  test:all              runs all the tests.
  test:integration      runs all the integration tests (use alias `mage integrationtest`).
  test:unit             runs all the unit tests (use alias `mage unitTest`).

Ex:
mage build:all = ENV PLATFORMS=all mage build builds all binaries
mage build builds binary for host machine
env PLATFORMS=windows mage build to build all binaries for windows
mage unitTest to run all unit tests

elasticmachine · 2022-07-06T13:35:17Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-08-23T13:10:42.640+0000
Duration: 12 min 54 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

.goreleaser.yaml

magefile.go

v1v · 2022-07-06T20:21:40Z

/test

ph

Added a few comments, but @narph this looks really good thanks for pushing theses changes!

ph · 2022-08-16T13:39:14Z

.ci/Jenkinsfile

-                  withGoEnv(){
-                    goTestJUnit(options: '-v ./...', output: 'junit-report.xml')
+                  withMageEnv(){
+                    cmd(label: 'Go unitTest', script: 'mage unitTest')
                  }


Not familiar with the withMageEnv call but I assume it generates the junit-report.xml ?

ph · 2022-08-16T13:45:57Z

.goreleaser.yaml

+      - amd64
+      - arm64
+    flags:
+      - -buildmode=pie


I believe we have more flags than that? like -s?

Question: Does goreleaser support creating universal binary or we have to have a post build hook to merge both?

we might be lucky using this https://goreleaser.com/customization/universalbinaries/

ph · 2022-08-16T13:47:56Z

.goreleaser.yaml

+      - '{{ if eq .Env.DEV "true" }}all=-N -l{{ end }}'
+changelog:
+  skip: true
+dist: build/binaries


This would be simple to share with different projects.

ph · 2022-08-16T13:58:38Z

dev-tools/common/test.go

+func GoUnitTest(ctx context.Context) error {
+	mg.Deps(InstallGoTestTools)
+
+	fmt.Println(">> go test:", "Unit Testing") //nolint:forbidigo // just for tests


I think if you are using a Print* methods and a writer, it should not mark any usage.

out := os.stdout fmt.Fprintln(out, >> go test:", "Unit Testing)

The above allow you to enable the output or not.

cmacknz · 2022-08-16T15:26:51Z

Running mage build:all now fails on my M1 Mac, trying to figure out if it's my machine or something in the build itself:

❯ mage build:all
>> build: Building binary for all
   • building...
   • loading config file       file=.goreleaser.yaml
   • loading environment variables
   • getting and validating git state
      • building...               commit=bdadcedcd64c77502e7b575303f836e8208e6b38 latest tag=v0.2.0
      • pipe skipped              error=validation is disabled
   • parsing tag
   • running before hooks
      • running                   hook=go mod tidy
      • running                   hook=go generate ./...
   • setting defaults
   • checking distribution directory
      • --rm-dist is set, cleaning it up
   • loading go mod information
   • build prerequisites
   • writing effective config file
      • writing                   config=build/binaries/config.yaml
   • building binaries
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-darwin-aarch64/elastic-agent-shipper
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-darwin-x86_64/elastic-agent-shipper
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-linux-x86_64/elastic-agent-shipper
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-linux-arm64/elastic-agent-shipper
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-linux-x86/elastic-agent-shipper
   ⨯ build failed after 2.76s error=failed to build for linux_386: exit status 2: # github.com/elastic/elastic-agent-shipper
loadinternal: cannot find runtime/cgo
/opt/homebrew/Cellar/go/1.18.4/libexec/pkg/tool/darwin_arm64/link: running clang failed: exit status 1
clang: warning: argument unused during compilation: '-pie' [-Wunused-command-line-argument]
ld: unknown/unsupported architecture name for: -arch armv4t
clang: error: linker command failed with exit code 1 (use -v to see invocation)

cmacknz · 2022-08-16T15:30:22Z

failed to build for linux_386 do we need to build for the 32-bit i386 arch? I don't think the agent supports 32 bit binaries right now.

ph · 2022-08-16T15:41:17Z

Afaik we don't need to support 32bits https://www.elastic.co/support/matrix

narph · 2022-08-22T09:40:58Z

Running mage build:all now fails on my M1 Mac, trying to figure out if it's my machine or something in the build itself:

❯ mage build:all
>> build: Building binary for all
   • building...
   • loading config file       file=.goreleaser.yaml
   • loading environment variables
   • getting and validating git state
      • building...               commit=bdadcedcd64c77502e7b575303f836e8208e6b38 latest tag=v0.2.0
      • pipe skipped              error=validation is disabled
   • parsing tag
   • running before hooks
      • running                   hook=go mod tidy
      • running                   hook=go generate ./...
   • setting defaults
   • checking distribution directory
      • --rm-dist is set, cleaning it up
   • loading go mod information
   • build prerequisites
   • writing effective config file
      • writing                   config=build/binaries/config.yaml
   • building binaries
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-darwin-aarch64/elastic-agent-shipper
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-darwin-x86_64/elastic-agent-shipper
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-linux-x86_64/elastic-agent-shipper
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-linux-arm64/elastic-agent-shipper
      • building                  binary=build/binaries/elastic-agent-shipper-8.5.0-linux-x86/elastic-agent-shipper
   ⨯ build failed after 2.76s error=failed to build for linux_386: exit status 2: # github.com/elastic/elastic-agent-shipper
loadinternal: cannot find runtime/cgo
/opt/homebrew/Cellar/go/1.18.4/libexec/pkg/tool/darwin_arm64/link: running clang failed: exit status 1
clang: warning: argument unused during compilation: '-pie' [-Wunused-command-line-argument]
ld: unknown/unsupported architecture name for: -arch armv4t
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@cmacknz , the build has cgo disabled (implicitly enabling static linking) and uses Go's native cross compilation which simplifies the builds.
This requires disabling -buildmode=pie for 386, which cannot be done except when using external linking, we do the same for fleet server builds (https://github.com/elastic/fleet-server/blob/main/Makefile#L8 build mode missing for linux/386), I've removed the flag so it should work now.

michalpristas · 2022-08-22T14:01:22Z

.gitignore


-
-# VSCode
-/.vscode


why viscose disappeared?

it's still there, in #directories

cmacknz · 2022-08-22T14:03:16Z

mage build
>> build: Building binary for
   • debug logs enabled
   • building...
   • loading config file       file=.goreleaser.yaml
   • loaded config file        config={ [] {   false false   [] []    } [] [] [] [] [] { {   }  { }      []  [] []} [{darwin [darwin] [amd64 arm64] [] [] [] []   {{ .ProjectName }}-{{ .Env.DEFAULT_VERSION }}-{{ .Os }}-{{ if eq .Arch "amd64" }}x86_64{{ end }}{{ if eq .Arch "arm64" }}aarch64{{ end }}/{{ .ProjectName }} {[] []} []   false  true   {[{{ if eq .Env.DEV "false" }}-s -w{{ end }} -X main.Version={{ .Env.DEFAULT_VERSION }} -X main.Commit={{.Commit}} -X main.BuildTime={{.Date}}] [] [-buildmode=pie] [] [{{ if eq .Env.DEV "true" }}all=-N -l{{ end }}]} []} {linux [linux] [amd64 arm64 386] [] [] [] []   {{ .ProjectName }}-{{ .Env.DEFAULT_VERSION }}-{{ .Os }}-{{ if eq .Arch "amd64" }}x86_64{{ end }}{{ if eq .Arch "386" }}x86{{ end }}{{ if eq .Arch "arm64" }}{{ .Arch }}{{ end }}/{{ .ProjectName }} {[] []} []   false  true   {[{{ if eq .Env.DEV "false" }}-s -w{{end}} -X main.Version={{ .Env.DEFAULT_VERSION }} -X main.Commit={{.Commit}} -X main.BuildTime={{.Date}}] [] [] [] [{{ if eq .Env.DEV "true" }}all=-N -l{{ end }}]} [{linux amd64   {[] [] [-buildmode=pie] [] []}} {linux arm   {[] [] [-buildmode=pie] [] []}}]} {windows [windows] [amd64 386] [] [] [] []   {{ .ProjectName }}-{{ .Env.DEFAULT_VERSION }}-{{ .Os }}-{{ if eq .Arch "amd64" }}x86_64{{ end }}{{ if eq .Arch "386" }}x86{{ end }}/{{ .ProjectName }} {[] []} []   false  true   {[{{ if eq .Env.DEV "false" }}-s -w{{ end }} -X main.Version={{ .Env.DEFAULT_VERSION }} -X main.Commit={{.Commit}} -X main.BuildTime={{.Date}}] [] [-buildmode=pie] [] [{{ if eq .Env.DEV "true" }}all=-N -l{{ end }}]} []}] [] [] [] {} {  [] false []} [] [] [] [] [] [] {{[]}  true  []} build/binaries [] [] {  } {[go mod tidy go generate ./...]} {  false } {false [] } { {false } {false     } {false     } {false    } {false    } {false  0   []   false} {false       } {false } {false  0} {false false   map[] }} [] [] { [] [] [] [] [] []    {[] []} []   false  false   {[] [] [] [] []} []} {   false} {  false false} {  false}}
   ⨯ build failed after 0.00s error=yaml: unmarshal errors:
  line 39: field goamd64 not found in type config.BuildDetailsOverride
  line 46: field goamd64 not found in type config.BuildDetailsOverride

The build is still failing for me, let me know if you need me to investigate anything.

michalpristas · 2022-08-22T14:10:14Z

dev-tools/common/common.go

+	if val := os.Getenv(buildEnv); val != "" {
+		boolVal, err := strconv.ParseBool(val)
+		if err != nil {
+			return defaultValue


can we shoot a warning here? maybe you want to know you provided incorrect value, may be a typo

dev-tools/common/test.go

michalpristas · 2022-08-22T14:11:23Z

dev-tools/common/test.go

+		)
+	}
+
+	testArgs = append(testArgs, []string{"./..."}...)


is windows ok with /?

michalpristas · 2022-08-22T14:13:56Z

magefile.go

-var (
-	goLicenserRepo = "github.com/elastic/go-licenser@v0.4.1"
+const (
+	GoreleaserRepo = "github.com/goreleaser/goreleaser@v1.6.3"


please ignore this if you don't like it, but I would separate .1.6.3 to a variable ending with Version so in case I forgot where it is defined I can guess based on full text search

michalpristas

few comments along the way, but overall it looks good

.goreleaser.yaml

AndersonQ · 2022-08-22T14:19:33Z

magefile.go

+		}
+		if isSnapshot {
+			args = append(args, "--snapshot")
+			env["DEFAULT_VERSION"] += fmt.Sprintf("-%s", "SNAPSHOT")


[Suggestion]
the fmt.Sprintf is unnecessary

Suggested change

env["DEFAULT_VERSION"] += fmt.Sprintf("-%s", "SNAPSHOT")

env["DEFAULT_VERSION"] += "-SNAPSHOT"

AndersonQ · 2022-08-22T14:20:45Z

magefile.go

+	return devtools.GoUnitTest(ctx, testCoverage)
+}
+
+//CHECKS


[NIT / Consistency]

Suggested change

//CHECKS

// CHECKS

AndersonQ · 2022-08-22T14:29:06Z

magefile.go

-}
-
-// CheckLicense checks the license headers
+// License should generate the license headers


[Suggestion]
Personally, I find better to have a function description to be assertive, it does either A or B, not it "should" do something.

Suggested change

// License should generate the license headers

// License generates the license headers or returns an error

narph · 2022-08-23T08:59:58Z

mage build
>> build: Building binary for
   • debug logs enabled
   • building...
   • loading config file       file=.goreleaser.yaml
   • loaded config file        config={ [] {   false false   [] []    } [] [] [] [] [] { {   }  { }      []  [] []} [{darwin [darwin] [amd64 arm64] [] [] [] []   {{ .ProjectName }}-{{ .Env.DEFAULT_VERSION }}-{{ .Os }}-{{ if eq .Arch "amd64" }}x86_64{{ end }}{{ if eq .Arch "arm64" }}aarch64{{ end }}/{{ .ProjectName }} {[] []} []   false  true   {[{{ if eq .Env.DEV "false" }}-s -w{{ end }} -X main.Version={{ .Env.DEFAULT_VERSION }} -X main.Commit={{.Commit}} -X main.BuildTime={{.Date}}] [] [-buildmode=pie] [] [{{ if eq .Env.DEV "true" }}all=-N -l{{ end }}]} []} {linux [linux] [amd64 arm64 386] [] [] [] []   {{ .ProjectName }}-{{ .Env.DEFAULT_VERSION }}-{{ .Os }}-{{ if eq .Arch "amd64" }}x86_64{{ end }}{{ if eq .Arch "386" }}x86{{ end }}{{ if eq .Arch "arm64" }}{{ .Arch }}{{ end }}/{{ .ProjectName }} {[] []} []   false  true   {[{{ if eq .Env.DEV "false" }}-s -w{{end}} -X main.Version={{ .Env.DEFAULT_VERSION }} -X main.Commit={{.Commit}} -X main.BuildTime={{.Date}}] [] [] [] [{{ if eq .Env.DEV "true" }}all=-N -l{{ end }}]} [{linux amd64   {[] [] [-buildmode=pie] [] []}} {linux arm   {[] [] [-buildmode=pie] [] []}}]} {windows [windows] [amd64 386] [] [] [] []   {{ .ProjectName }}-{{ .Env.DEFAULT_VERSION }}-{{ .Os }}-{{ if eq .Arch "amd64" }}x86_64{{ end }}{{ if eq .Arch "386" }}x86{{ end }}/{{ .ProjectName }} {[] []} []   false  true   {[{{ if eq .Env.DEV "false" }}-s -w{{ end }} -X main.Version={{ .Env.DEFAULT_VERSION }} -X main.Commit={{.Commit}} -X main.BuildTime={{.Date}}] [] [-buildmode=pie] [] [{{ if eq .Env.DEV "true" }}all=-N -l{{ end }}]} []}] [] [] [] {} {  [] false []} [] [] [] [] [] [] {{[]}  true  []} build/binaries [] [] {  } {[go mod tidy go generate ./...]} {  false } {false [] } { {false } {false     } {false     } {false    } {false    } {false  0   []   false} {false       } {false } {false  0} {false false   map[] }} [] [] { [] [] [] [] [] []    {[] []} []   false  false   {[] [] [] [] []} []} {   false} {  false false} {  false}}
   ⨯ build failed after 0.00s error=yaml: unmarshal errors:
  line 39: field goamd64 not found in type config.BuildDetailsOverride
  line 46: field goamd64 not found in type config.BuildDetailsOverride

The build is still failing for me, let me know if you need me to investigate anything.

I ran locally with a recent version of goreleaser which contains goamd64 field in BuildDetailsOverride schema, removed it as it made no difference to the logic, can you try again?

cmacknz · 2022-08-23T13:56:21Z

Build is fixed for me, thanks!

narph added 3 commits July 6, 2022 13:53

trial

f58d6a9

merge

bdea14a

change

51ba2db

narph requested a review from a team as a code owner July 6, 2022 13:24

narph requested review from cmacknz and fearful-symmetry and removed request for a team July 6, 2022 13:24

narph self-assigned this Jul 6, 2022

narph marked this pull request as draft July 6, 2022 13:24

narph requested a review from v1v July 6, 2022 13:25

narph added the Team:Elastic-Agent Label for the Agent team label Jul 6, 2022

narph mentioned this pull request Jul 6, 2022

Create build and release pipelines for the elastic agent shipper #74

Closed

4 tasks

v1v reviewed Jul 6, 2022

View reviewed changes

.goreleaser.yaml Outdated Show resolved Hide resolved

.goreleaser.yaml Outdated Show resolved Hide resolved

.goreleaser.yaml Outdated Show resolved Hide resolved

.goreleaser.yaml Outdated Show resolved Hide resolved

narph added 2 commits July 6, 2022 16:14

add pkg

80e84a7

update

e082b45

mdelapenya reviewed Jul 6, 2022

View reviewed changes

magefile.go Outdated Show resolved Hide resolved

update

6968cde

v1v mentioned this pull request Jul 6, 2022

Add support for go packages with explicit versions elastic/apm-pipeline-library#1764

Merged

narph added 10 commits July 13, 2022 15:26

work on mage

d5f247e

work on mage

5a9df25

work on ci

b984b1d

work on ci

d9034f7

fix

b83be8a

remove

b85bb5b

notice

e480da7

forbidigo

7d56bcf

forbidigo

a6520ef

sort imports

a5d752b

ph reviewed Aug 16, 2022

View reviewed changes

cmacknz requested review from rdner and faec August 16, 2022 15:22

version update

51c7c40

narph requested a review from michalpristas August 22, 2022 09:41

michalpristas reviewed Aug 22, 2022

View reviewed changes

dev-tools/common/test.go Outdated Show resolved Hide resolved

michalpristas reviewed Aug 22, 2022

View reviewed changes

dev-tools/common/test.go

)

}

testArgs = append(testArgs, []string{"./..."}...)

Copy link

Contributor

michalpristas Aug 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is windows ok with /?

michalpristas reviewed Aug 22, 2022

View reviewed changes

michalpristas approved these changes Aug 22, 2022

View reviewed changes

AndersonQ reviewed Aug 22, 2022

View reviewed changes

narph added 2 commits August 23, 2022 10:49

version fix

223c098

merge

abff0ba

narph added 5 commits August 23, 2022 11:18

lint

160f981

add binary check

81b1e6e

add binary check

981a418

remove

fb3f91a

rename env

a26ab2f

cmacknz approved these changes Aug 23, 2022

View reviewed changes

narph merged commit b855ef3 into elastic:main Aug 23, 2022

narph deleted the add-build branch August 23, 2022 14:16

cmacknz mentioned this pull request Dec 19, 2024

Investigate using GoReleaser to package the Elastic Agent elastic/elastic-agent#6416

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expand mage and add build options #73

Expand mage and add build options #73

narph commented Jul 6, 2022 •

edited

Loading

elasticmachine commented Jul 6, 2022 •

edited

Loading

Build stats

v1v commented Jul 6, 2022

ph left a comment

ph Aug 16, 2022

ph Aug 16, 2022

ph Aug 16, 2022

narph Aug 16, 2022

ph Aug 16, 2022

ph Aug 16, 2022

cmacknz commented Aug 16, 2022

cmacknz commented Aug 16, 2022

ph commented Aug 16, 2022

narph commented Aug 22, 2022

michalpristas Aug 22, 2022

narph Aug 23, 2022

cmacknz commented Aug 22, 2022

michalpristas Aug 22, 2022

michalpristas Aug 22, 2022

michalpristas Aug 22, 2022 •

edited

Loading

michalpristas left a comment

AndersonQ Aug 22, 2022

AndersonQ Aug 22, 2022

AndersonQ Aug 22, 2022

narph commented Aug 23, 2022

cmacknz commented Aug 23, 2022

	env["DEFAULT_VERSION"] += fmt.Sprintf("-%s", "SNAPSHOT")
	env["DEFAULT_VERSION"] += "-SNAPSHOT"

	// License should generate the license headers
	// License generates the license headers or returns an error



		# VSCode
		/.vscode

Expand mage and add build options #73

Expand mage and add build options #73

Conversation

narph commented Jul 6, 2022 • edited Loading

elasticmachine commented Jul 6, 2022 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

v1v commented Jul 6, 2022

ph left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmacknz commented Aug 16, 2022

cmacknz commented Aug 16, 2022

ph commented Aug 16, 2022

narph commented Aug 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmacknz commented Aug 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalpristas Aug 22, 2022 • edited Loading

Choose a reason for hiding this comment

michalpristas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narph commented Aug 23, 2022

cmacknz commented Aug 23, 2022

narph commented Jul 6, 2022 •

edited

Loading

elasticmachine commented Jul 6, 2022 •

edited

Loading

michalpristas Aug 22, 2022 •

edited

Loading