[CI] enable Windows-2016 and support 32 bits build requirements #19199

v1v · 2020-06-15T15:09:43Z

What does this PR do?

Configure the tool dependencies that are required (the workers might have some misconfiguration in place unfortunately while it happens we need to use this approach):
- Install gvm 32 bits if not installed in the build scripts.
- gvm fallback to 32 bits when running on a Windows 32 bit in the CI Pipeline.
- Install gcc in the build scripts.
Enable Windows-2019 and Windows-2016 only
Support 32 bits Windows in the mage build system.

Why is it important?

As long as we are working on enabling the Windows CI Workers then we need to avoid the overkilling of spinning too many VMs when the thing is broken. Fix certain dependencies that are required in the workers but we don't want to be. blocked until they are fixed.

Tasks

Test failures only for filebeat in Window-2016. [filebeat] in windows-2016 got some flakiness tests for TestReceiveEventsAndMetadata #19641. Disabled for the time being.

elasticmachine · 2020-06-15T15:20:38Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #19199 updated]
Start Time: 2020-07-03T14:08:39.913+0000
Duration: 74 min 4 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9747
Skipped	1579
Total	11326

Steps errors

Expand to view the steps failures

Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 2 min 22 sec
- Start Time: 2020-07-03T15:10:48.896+0000
- log

This reverts commit f1715e7.

This reverts commit 538ede0.

This reverts commit d4788d6.

…e time being

v1v · 2020-06-17T11:18:34Z

.ci/scripts/install-tools.bat

+REM If 32 bits then install the GVM accordingly
+IF NOT EXIST "%PROGRAMFILES(X86)%" (
+    curl -sL -o %WORKSPACE%\bin\gvm.exe https://github.com/andrewkroh/gvm/releases/download/v0.2.2/gvm-windows-386.exe
+)
+


NOTE: this could be removed as soon as the packer images have been deployed with the 32 bit support

v1v · 2020-06-17T16:06:26Z

.ci/scripts/install-tools.bat

@@ -1,15 +1,25 @@
 set GOPATH=%WORKSPACE%
 set MAGEFILE_CACHE=%WORKSPACE%\.magefile
-set PATH=%WORKSPACE%\bin;C:\ProgramData\chocolatey\bin;%PATH%
+set PATH=%WORKSPACE%\bin;C:\ProgramData\chocolatey\bin;C:\tools\mingw64\bin;%PATH%


Where the installation of the gcc is done.

NOTE: this could be removed as soon as the packer images have been deployed with support to gcc.

v1v · 2020-06-17T16:06:41Z

.ci/windows.groovy

    "GOPATH=${env.WORKSPACE}",
    "GOROOT=${goRoot}",
-    "PATH=${env.WORKSPACE}\\bin;${goRoot}\\bin;${chocoPath};${chocoPython3Path};${env.PATH}",
+    "PATH=${env.WORKSPACE}\\bin;${goRoot}\\bin;${chocoPath};${chocoPython3Path};C:\\tools\\mingw64\\bin;${env.PATH}",


NOTE: this could be removed as soon as the packer images have been deployed with support to gcc.

v1v · 2020-06-17T16:08:30Z

dev-tools/mage/gotest.go

+	// -race is only supported on */amd64
+	if os.Getenv("DEV_ARCH") == "amd64" {
+		if params.Race {
+			args = append(args, "-race")
+		}


I don't know if we could use a different approach. The current issue happens when running in a CI Windows worker the mage dumpVariables reports the environment variable GOARCH and GOOS accordingly, but when running in the same CI windows worker the command mage -v build unitTest those variables are pointing to something else Adding build environment vars: map[CGO_ENABLED:0 GOARCH:amd64 GOFLAGS:-mod=vendor GOOS:linux]

v1v · 2020-06-17T16:08:41Z

x-pack/elastic-agent/magefile.go

@@ -234,7 +234,11 @@ func (Test) All() {
 // Unit runs all the unit tests.
 func (Test) Unit() error {
 	mg.Deps(Prepare.Env, Build.TestBinaries)
-	return RunGo("test", "-race", "-v", "-coverprofile", filepath.Join(buildDir, "coverage.out"), "./...")
+	raceFlag := ""


See https://github.com/elastic/beats/pull/19199/files#r441661503

v1v · 2020-06-17T16:08:55Z

.ci/windows.groovy

+    "DEV_ARCH=${arch}",
+    "DEV_OS=windows",


See https://github.com/elastic/beats/pull/19199/files#r441661503

…ne-beats * upstream/master: (105 commits) ci: enable packaging job (elastic#19536) ci: disable upstream trigger on PRs for the packaging job (elastic#19490) Implement memlog on-disk handling (elastic#19408) fix go.mod for PR elastic#19423 (elastic#19521) [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions (elastic#19423) Input v2 stateless manager (elastic#19406) Input v2 compatibility layer (elastic#19401) [Elastic Agent] Fix artifact downloading to allow endpoint-security to be downloaded (elastic#19503) fix: ignore target changes on scans (elastic#19510) Add more helpers to pipeline/testing package (elastic#19405) Report dependencies in CSV format (elastic#19506) [Filebeat] Fix reference leak in TCP and Unix socket inputs (elastic#19459) Cursor input skeleton (elastic#19378) Add changelog. (elastic#19495) [DOC] Typo in Kerberos (elastic#19265) Remove accidentally commited unused NOTICE template (elastic#19485) [Elastic Agent] Support the install, control, and uninstall of Endpoint (elastic#19248) [Filebeat][httpjson] Add split_events_by config setting (elastic#19246) ci: disabling packaging job until we fix it (elastic#19481) Fix golang.org/x/tools to release1.13 (elastic#19478) ...

more verbose output

…ne-beats * upstream/master: (35 commits) [ci] fix env variable name for xpack filebeats (elastic#19617) Cache error responses for cloudfoundry apps metadata (elastic#19181) ci: user fixed type of agent (elastic#19625) Input v2 cursor testing (elastic#19573) Update Jenkinsfile to not inspect removed vendor (elastic#19610) Fix ordering and duplicate configs on autodiscover (elastic#19317) Prepare input/file for changes in the registrar (elastic#19516) Cursor input and manager implementation (elastic#19571) [Filebeat] Fix tls mapping in suricata module (elastic#19494) [Ingest Manager] Make Agent beta and Constraints experimental (elastic#19586) Accept prefix as metric_types for stackdriver metricset in GCP (elastic#19345) Implement memlog store operations (elastic#19533) introduce journalbeat/pkg in order to provide reusable shared code (elastic#19581) Add descriptions to HAProxy fields in Metricbeat (elastic#19561) ci: apm-server-update trigered only on upstream, comments, and manual triggered (elastic#19590) ci: enable upstream triggering on the packaging job (elastic#19589) ci: some jjbb improvements (elastic#19588) [MetricBeat] set tags correctly if the dimension value is ARN (elastic#19433) [Filebeat] Add default_fields: false to fields.yml in aws module (elastic#19568) Add publisher implementation for stateful inputs (elastic#19530) ...

elasticmachine · 2020-07-03T14:09:35Z

Pinging @elastic/ingest-management (Team:Ingest Management)

mdelapenya · 2020-07-03T16:05:02Z

.ci/windows.groovy

-          bat(label: "Mage ${target}", script: "mage ${verboseFlag} ${target}")
+    // NOTE: skip filebeat with windows-2016 since there are some test failures.
+    if (directory.equals('filebeat') && label.equals('windows-2016')) {
+      log(level: 'WARN', text: "Skipped stage for the 'filebeat' with 'windows-2016' as long as there are test failures to be analysed.")


I'd prefer having the build red because of those failures, so that the team can work on them to fix the build. But I'm ok with this change :)

…tic#19199)

…tic#19199) # Conflicts: # .ci/windows.groovy # x-pack/elastic-agent/magefile.go

…) (#22348)

…) (#22347)

[CI] enable one beats for windows

0704fc5

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 15, 2020

this is required to set the GO_VERSION

40b96b2

v1v added Team:Automation Label for the Observability productivity team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 15, 2020

v1v self-assigned this Jun 15, 2020

v1v added 10 commits June 15, 2020 17:02

support one stage

d901d22

chore: for testing purposes

f1715e7

install gvm 32 bits if required

260c0bd

Revert "chore: for testing purposes"

78daec7

This reverts commit f1715e7.

chore: for testing purposes

d4788d6

chore: cosmetic change

ed3874b

ci: support 32 bits for gvm

538ede0

Revert "ci: support 32 bits for gvm"

bcaf8b0

This reverts commit 538ede0.

ci: support 32 bits for gvm

c009149

amd64 GOARCH and --race only on amd64

8d8bba8

v1v force-pushed the feature/windows-one-beats branch from f305f0e to 8d8bba8 Compare June 17, 2020 09:20

v1v added 8 commits June 17, 2020 10:28

DEV_OS=windows

648362d

Revert "chore: for testing purposes"

ac237dd

This reverts commit d4788d6.

x-pack/elastic-agent

fdfffe1

unrequired

ce6ce42

amd64 GOARCH and --race only on amd64

dcb6801

gcc is not available in the w2012-r2 workers, let's disable it for th…

0e3a1f1

…e time being

enable all the remaining stages

f6803b1

fix typo

f1d2390

v1v commented Jun 17, 2020

View reviewed changes

v1v added 3 commits June 17, 2020 14:18

Unique ID

20160f8

ci: disable w10 and enable Functionbeat

e765547

gcc installation

32ae6bf

v1v commented Jun 17, 2020

View reviewed changes

v1v changed the title ~~[CI] enable one beats for windows~~ [CI] enable Windows-2016 and support 32 bits build requirements Jul 2, 2020

v1v mentioned this pull request Jul 3, 2020

[filebeat] in windows-2016 got some flakiness tests for TestReceiveEventsAndMetadata #19641

Closed

ci: skip filebeat for windows-2016 since there are test failures

1ee8f70

more verbose output

v1v force-pushed the feature/windows-one-beats branch from 4efad36 to 1ee8f70 Compare July 3, 2020 14:05

v1v marked this pull request as ready for review July 3, 2020 14:09

v1v requested a review from a team as a code owner July 3, 2020 14:09

botelastic bot added the Team:Ingest Management label Jul 3, 2020

v1v requested review from andrewkroh and ph July 3, 2020 14:09

mdelapenya reviewed Jul 3, 2020

View reviewed changes

mdelapenya approved these changes Jul 3, 2020

View reviewed changes

cachedout approved these changes Jul 6, 2020

View reviewed changes

ph approved these changes Jul 7, 2020

View reviewed changes

v1v merged commit 7dc1fcd into elastic:master Jul 8, 2020

v1v deleted the feature/windows-one-beats branch July 8, 2020 15:42

melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020

[CI] enable Windows-2016 and support 32 bits build requirements (elas…

307c212

…tic#19199)

v1v added a commit to v1v/beats that referenced this pull request Nov 2, 2020

[CI] enable Windows-2016 and support 32 bits build requirements (elas…

35bc77e

…tic#19199)

v1v mentioned this pull request Nov 2, 2020

Cherry-pick to 7.x: [CI] enable Windows-2016 and support 32 bits build requirements (#19199) #22347

Merged

v1v added a commit to v1v/beats that referenced this pull request Nov 2, 2020

[CI] enable Windows-2016 and support 32 bits build requirements (elas…

a456c45

…tic#19199) # Conflicts: # .ci/windows.groovy # x-pack/elastic-agent/magefile.go

v1v mentioned this pull request Nov 2, 2020

Cherry-pick to 7.10: [CI] enable Windows-2016 and support 32 bits build requirements (#19199) #22348

Merged

v1v added a commit that referenced this pull request Nov 2, 2020

[CI] enable Windows-2016 and support 32 bits build requirements (#19199…

54623f7

…) (#22348)

v1v added a commit that referenced this pull request Nov 2, 2020

[CI] enable Windows-2016 and support 32 bits build requirements (#19199…

d79de9d

…) (#22347)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] enable Windows-2016 and support 32 bits build requirements #19199

[CI] enable Windows-2016 and support 32 bits build requirements #19199

v1v commented Jun 15, 2020 •

edited

Loading

elasticmachine commented Jun 15, 2020 •

edited

Loading

Build stats

Test stats 🧪

v1v Jun 17, 2020

v1v Jun 17, 2020

v1v Jun 17, 2020

v1v Jun 17, 2020

v1v Jun 17, 2020

v1v Jun 17, 2020

elasticmachine commented Jul 3, 2020

mdelapenya Jul 3, 2020

[CI] enable Windows-2016 and support 32 bits build requirements #19199

[CI] enable Windows-2016 and support 32 bits build requirements #19199

Conversation

v1v commented Jun 15, 2020 • edited Loading

What does this PR do?

Why is it important?

Tasks

elasticmachine commented Jun 15, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

v1v Jun 17, 2020

Choose a reason for hiding this comment

v1v Jun 17, 2020

Choose a reason for hiding this comment

v1v Jun 17, 2020

Choose a reason for hiding this comment

v1v Jun 17, 2020

Choose a reason for hiding this comment

v1v Jun 17, 2020

Choose a reason for hiding this comment

v1v Jun 17, 2020

Choose a reason for hiding this comment

elasticmachine commented Jul 3, 2020

mdelapenya Jul 3, 2020

Choose a reason for hiding this comment

v1v commented Jun 15, 2020 •

edited

Loading

elasticmachine commented Jun 15, 2020 •

edited

Loading