Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] enable Windows-2016 and support 32 bits build requirements #19199

Merged
merged 30 commits into from
Jul 8, 2020
Merged

[CI] enable Windows-2016 and support 32 bits build requirements #19199

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
0704fc5
[CI] enable one beats for windows
v1v Jun 15, 2020
40b96b2
this is required to set the GO_VERSION
v1v Jun 15, 2020
d901d22
support one stage
v1v Jun 15, 2020
f1715e7
chore: for testing purposes
v1v Jun 16, 2020
260c0bd
install gvm 32 bits if required
v1v Jun 16, 2020
78daec7
Revert "chore: for testing purposes"
v1v Jun 16, 2020
d4788d6
chore: for testing purposes
v1v Jun 16, 2020
ed3874b
chore: cosmetic change
v1v Jun 16, 2020
538ede0
ci: support 32 bits for gvm
v1v Jun 17, 2020
bcaf8b0
Revert "ci: support 32 bits for gvm"
v1v Jun 17, 2020
c009149
ci: support 32 bits for gvm
v1v Jun 17, 2020
8d8bba8
amd64 GOARCH and --race only on amd64
v1v Jun 17, 2020
648362d
DEV_OS=windows
v1v Jun 17, 2020
ac237dd
Revert "chore: for testing purposes"
v1v Jun 17, 2020
fdfffe1
x-pack/elastic-agent
v1v Jun 17, 2020
ce6ce42
unrequired
v1v Jun 17, 2020
dcb6801
amd64 GOARCH and --race only on amd64
v1v Jun 17, 2020
0e3a1f1
gcc is not available in the w2012-r2 workers, let's disable it for th…
v1v Jun 17, 2020
f6803b1
enable all the remaining stages
v1v Jun 17, 2020
f1d2390
fix typo
v1v Jun 17, 2020
20160f8
Unique ID
v1v Jun 17, 2020
e765547
ci: disable w10 and enable Functionbeat
v1v Jun 17, 2020
32ae6bf
gcc installation
v1v Jun 17, 2020
e05cdb9
enable windows-2012-r2 to validate the gcc instalation
v1v Jun 17, 2020
44f3b83
fix gcc validations
v1v Jun 17, 2020
802a622
Add gcc to the path
v1v Jun 17, 2020
d74b1b7
Subset of supported versions until we make them to work
v1v Jun 17, 2020
c57b3ae
Merge remote-tracking branch 'upstream/master' into feature/windows-o…
v1v Jul 2, 2020
1ee8f70
ci: skip filebeat for windows-2016 since there are test failures
v1v Jul 2, 2020
0ba6f3b
Merge remote-tracking branch 'upstream/master' into feature/windows-o…
v1v Jul 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions .ci/scripts/install-tools.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,25 @@
set GOPATH=%WORKSPACE%
set MAGEFILE_CACHE=%WORKSPACE%\.magefile
set PATH=%WORKSPACE%\bin;C:\ProgramData\chocolatey\bin;%PATH%
set PATH=%WORKSPACE%\bin;C:\ProgramData\chocolatey\bin;C:\tools\mingw64\bin;%PATH%
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where the installation of the gcc is done.

NOTE: this could be removed as soon as the packer images have been deployed with support to gcc.


where /q curl
IF ERRORLEVEL 1 (
choco install curl -y --no-progress --skipdownloadcache
)
mkdir %WORKSPACE%\bin

REM If 32 bits then install the GVM accordingly
IF NOT EXIST "%PROGRAMFILES(X86)%" (
curl -sL -o %WORKSPACE%\bin\gvm.exe https://github.com/andrewkroh/gvm/releases/download/v0.2.2/gvm-windows-386.exe
)

Comment on lines +11 to +15
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: this could be removed as soon as the packer images have been deployed with the 32 bit support

where /q gvm
IF ERRORLEVEL 1 (
curl -sL -o %WORKSPACE%\bin\gvm.exe https://github.com/andrewkroh/gvm/releases/download/v0.2.2/gvm-windows-amd64.exe
IF EXIST "%PROGRAMFILES(X86)%" (
curl -sL -o %WORKSPACE%\bin\gvm.exe https://github.com/andrewkroh/gvm/releases/download/v0.2.2/gvm-windows-amd64.exe
) ELSE (
curl -sL -o %WORKSPACE%\bin\gvm.exe https://github.com/andrewkroh/gvm/releases/download/v0.2.2/gvm-windows-386.exe
)
)
FOR /f "tokens=*" %%i IN ('"gvm.exe" use %GO_VERSION% --format=batch') DO %%i

Expand All @@ -25,3 +35,9 @@ if not exist C:\Python38\python.exe (
python --version
where python

if not exist C:\tools\mingw64\bin\gcc.exe (
REM Install mingw 5.3.0
choco install mingw -y -r --no-progress --version 5.3.0 || echo ERROR && exit /b
)
gcc --version
where gcc
58 changes: 38 additions & 20 deletions .ci/windows.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,12 @@ import groovy.transform.Field

/**
List of supported windows versions to be tested with
NOTE:
- 'windows-10' is too slow
- 'windows-2012-r2', 'windows-2008-r2', 'windows-7', 'windows-7-32-bit' are disabled
since we are working on releasing each windows version incrementally.
*/
@Field def windowsVersions = ['windows-2019', 'windows-2016', 'windows-2012-r2', 'windows-10', 'windows-2008-r2', 'windows-7', 'windows-7-32-bit']
@Field def windowsVersions = ['windows-2019', 'windows-2016']

pipeline {
agent { label 'ubuntu && immutable' }
Expand Down Expand Up @@ -66,9 +70,7 @@ pipeline {
gitCheckout(basedir: "${BASE_DIR}", githubNotifyFirstTimeContributor: true)
stashV2(name: 'source', bucket: "${JOB_GCS_BUCKET}", credentialsId: "${JOB_GCS_CREDENTIALS}")
dir("${BASE_DIR}"){
// NOTE: commented to run faster the windows pipeline.
// when required then it can be enabled.
// loadConfigEnvVars()
loadConfigEnvVars()
}
whenTrue(params.debug){
dumpFilteredEnvironment()
Expand Down Expand Up @@ -246,20 +248,21 @@ pipeline {
}
}
steps {
mageTargetWin("Winlogbeat Windows Unit test", "x-pack/winlogbeat", "build unitTest")
mageTargetWin("Winlogbeat x-pack Windows", "x-pack/winlogbeat", "build unitTest")
}
}
stage('Functionbeat'){
options { skipDefaultCheckout() }
when {
beforeAgent true
expression {
return env.BUILD_FUNCTIONBEAT_XPACK != "false"
return params.windowsTest
// NOTE: commented to run all the windows stages.
//return env.BUILD_FUNCTIONBEAT_XPACK != "false"
}
}
stages {
stage('Functionbeat Windows'){
agent { label 'windows-immutable && windows-2019' }
stage('Functionbeat Windows x-pack'){
options { skipDefaultCheckout() }
when {
beforeAgent true
Expand All @@ -268,7 +271,7 @@ pipeline {
}
}
steps {
mageTargetWin("Functionbeat Windows Unit test", "x-pack/functionbeat", "build unitTest")
mageTargetWin("Functionbeat x-pack Windows Unit test", "x-pack/functionbeat", "build unitTest")
}
}
}
Expand Down Expand Up @@ -345,16 +348,22 @@ def mageTargetWin(String context, String directory, String target, String label)
return {
log(level: 'INFO', text: "context=${context} directory=${directory} target=${target} os=${label}")
def immutable = label.equals('windows-7-32-bit') ? 'windows-immutable-32-bit' : 'windows-immutable'
node("${immutable} && ${label}"){
withBeatsEnvWin() {
whenTrue(params.debug) {
dumpFilteredEnvironment()
dumpMageWin()
}

def verboseFlag = params.debug ? "-v" : ""
dir(directory) {
bat(label: "Mage ${target}", script: "mage ${verboseFlag} ${target}")
// NOTE: skip filebeat with windows-2016 since there are some test failures.
if (directory.equals('filebeat') && label.equals('windows-2016')) {
log(level: 'WARN', text: "Skipped stage for the 'filebeat' with 'windows-2016' as long as there are test failures to be analysed.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer having the build red because of those failures, so that the team can work on them to fix the build. But I'm ok with this change :)

} else {
node("${immutable} && ${label}"){
withBeatsEnvWin() {
whenTrue(params.debug) {
dumpFilteredEnvironment()
dumpMageWin()
}

def verboseFlag = params.debug ? "-v" : ""
dir(directory) {
bat(label: "Mage ${target}", script: "mage ${verboseFlag} ${target}")
}
}
}
}
Expand Down Expand Up @@ -407,13 +416,17 @@ def withBeatsEnv(boolean archive, Closure body) {
def withBeatsEnvWin(Closure body) {
final String chocoPath = 'C:\\ProgramData\\chocolatey\\bin'
final String chocoPython3Path = 'C:\\Python38;C:\\Python38\\Scripts'
def goRoot = "${env.USERPROFILE}\\.gvm\\versions\\go${GO_VERSION}.windows.amd64"
// NOTE: to support Windows 7 32 bits the arch in the go context path is required.
def arch = is32bit() ? '386' : 'amd64'
def goRoot = "${env.USERPROFILE}\\.gvm\\versions\\go${GO_VERSION}.windows.${arch}"

withEnv([
"HOME=${env.WORKSPACE}",
"DEV_ARCH=${arch}",
"DEV_OS=windows",
Comment on lines +425 to +426
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://github.com/elastic/beats/pull/19199/files#r441661503

"GOPATH=${env.WORKSPACE}",
"GOROOT=${goRoot}",
"PATH=${env.WORKSPACE}\\bin;${goRoot}\\bin;${chocoPath};${chocoPython3Path};${env.PATH}",
"PATH=${env.WORKSPACE}\\bin;${goRoot}\\bin;${chocoPath};${chocoPython3Path};C:\\tools\\mingw64\\bin;${env.PATH}",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: this could be removed as soon as the packer images have been deployed with support to gcc.

"MAGEFILE_CACHE=${env.WORKSPACE}\\.magefile",
"TEST_COVERAGE=true",
"RACE_DETECTOR=true",
Expand Down Expand Up @@ -448,6 +461,11 @@ def installTools() {
}
}

def is32bit(){
def labels = env.NODE_LABELS
return labels.contains('i386')
}

def goos(){
def labels = env.NODE_LABELS

Expand Down
7 changes: 5 additions & 2 deletions dev-tools/mage/gotest.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,8 +193,11 @@ func GoTest(ctx context.Context, params GoTestArgs) error {
args := []string{"test"}
args = append(args, "-v")

if params.Race {
args = append(args, "-race")
// -race is only supported on */amd64
if os.Getenv("DEV_ARCH") == "amd64" {
if params.Race {
args = append(args, "-race")
}
Comment on lines +196 to +200
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if we could use a different approach. The current issue happens when running in a CI Windows worker the mage dumpVariables reports the environment variable GOARCH and GOOS accordingly, but when running in the same CI windows worker the command mage -v build unitTest those variables are pointing to something else Adding build environment vars: map[CGO_ENABLED:0 GOARCH:amd64 GOFLAGS:-mod=vendor GOOS:linux]

}
if len(params.Tags) > 0 {
args = append(args, "-tags", strings.Join(params.Tags, " "))
Expand Down
6 changes: 5 additions & 1 deletion x-pack/elastic-agent/magefile.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,11 @@ func (Test) All() {
// Unit runs all the unit tests.
func (Test) Unit() error {
mg.Deps(Prepare.Env, Build.TestBinaries)
return RunGo("test", "-race", "-v", "-coverprofile", filepath.Join(buildDir, "coverage.out"), "./...")
raceFlag := ""
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://github.com/elastic/beats/pull/19199/files#r441661503

if os.Getenv("DEV_ARCH") == "amd64" {
raceFlag = "-race"
}
return RunGo("test", raceFlag, "-v", "-coverprofile", filepath.Join(buildDir, "coverage.out"), "./...")
}

// Coverage takes the coverages report from running all the tests and display the results in the browser.
Expand Down