Jenkins test suites are disabled because fails

[enricosada]

current Jenkins build fails. /cc @otawfik-ms @KevinRansom

it's ok, but because is merged, all new pull request fails, and that a bit annoying.

It's possible to:

• make master green, removing the groove file or doing compilation only ( appveyor-build.cmd all,build_only )
• fixed in a pull request
• merged after is green

QASuite

example log: http://dotnet-ci.cloudapp.net/job/Microsoft_visualfsharp/job/QASuite_prtest/8/console

update.cmd release -ngen fails:

• Failed to open registry key -- Administrator permissions are needed
• ngen fail: Administrator permissions are needed to use the selected options
• gac Failure adding assembly to the cache: Administrator permissions are needed

test fails because cannot find FSharp.Core 4.4.1 ( System.IO.FileNotFoundException: Could not load file or assembly 'FSharp.Core, Version=4.4.1.9055, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' )
I think that's expected because it failed to install in gac

UnitTests

example log: http://dotnet-ci.cloudapp.net/job/Microsoft_visualfsharp/job/UnitTests_prtest/8/

the flacky test Async.AwaitWaitHandle does not leak memory ( ref #811 )

06:35:52 1) Failed : FSharp.Core.Unittests.FSharp_Core.Microsoft_FSharp_Control.AsyncModule.Async.AwaitWaitHandle does not leak memory
06:35:52   Expected: False
06:35:52   But was:  True

CambridgeTests

example log: http://dotnet-ci.cloudapp.net/job/Microsoft_visualfsharp/job/CambridgeTests_prtest/8/console

same problems of QASuite, the gac install fails, and build timeout ( maybe because some test wait because gac'ed versions doesnt exists )

[OmarTawfik]

I don't think we should disable Jenkins, for (1) we are currently relying on appveyor for our CI efforts, and (2) we are incrementally bringing jenkins to par. We can look into each of these issues until it is on the same level/predictability as appveyor.

For GAC failures, @mmitche our build scripts need admin permissions. Is this possible?
@enricosada we can look into test failures after this is fixed.

[mmitche]

There is one machine that has elevated permissions right now (windows-elevated is the label I think). It is possible to get around using the gac and altering global state?

Matt

From: Omar Tawfik [mailto:notifications@github.com]
Sent: Friday, January 8, 2016 11:24 AM
To: Microsoft/visualfsharp visualfsharp@noreply.github.com
Cc: Matt Mitchell mmitche@microsoft.com
Subject: Re: [visualfsharp] Jenkins build fails (#859)
Importance: High

I don't think we should disable Jenkins, for (1) we are currently relying on appveyor for our CI efforts, and (2) we are incrementally bringing jenkins to par. We can look into each of these issues until it is on the same level/predictability as appveyor.

For GAC failures, @mmitchehttps://github.com/mmitche our build scripts need admin permissions. Is this possible?
@enricosadahttps://github.com/enricosada we can look into test failures after this is fixed.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/859#issuecomment-170098893.

[enricosada]

I'd like to have jenkins.
What i mean is, now only compile works, so do compile only.
And fix errors inside a pull requests so the others doesnt have red build.
It's annoying

You can have a single build job "WindowsRelease" , in master does compile only all,build_only
In the pr you do all,smoke and you can fix errors

The gac is also a problem. It's not possible to run concurrent jobs with shared gac, so need to exists a limit

[mmitche]

We don’t run concurrent on Windows machines for other reasons (msbuild process reuse for one), so we’re at least good there.

From: Enrico Sada [mailto:notifications@github.com]
Sent: Friday, January 8, 2016 11:48 AM
To: Microsoft/visualfsharp visualfsharp@noreply.github.com
Cc: Matt Mitchell mmitche@microsoft.com
Subject: Re: [visualfsharp] Jenkins build fails (#859)

I'd like to have jenkins.
What i mean is, now only compile works, so do compile only.
And fix errors inside a pull requests so the others doesnt have red build.
It's annoying

You can have a single build job "WindowsRelease" , in master does compile only all,build_only
In the pr you do all,smoke and you can fix errors

The gac is also a problem. It's not possible to run concurrent jobs with shared gac, so need to exists a limit

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/859#issuecomment-170105118.

[OmarTawfik]

@enricosada I agree it is annoying, but we cannot keep Jenkins running tests in a PR, as the job definitions are regenerated on the CI server, not pulled from every commit. I'm planning on doing work on the GAC items in the next period, so this should be resolved.

@mmitche any pointers on using the elevated-permissions machine for now?

[mmitche]

Change the label in your netci.groovy file from “windows” to “windows-elevated”

Matt

From: Omar Tawfik [mailto:notifications@github.com]
Sent: Friday, January 8, 2016 12:36 PM
To: Microsoft/visualfsharp visualfsharp@noreply.github.com
Cc: Matt Mitchell mmitche@microsoft.com
Subject: Re: [visualfsharp] Jenkins build fails (#859)
Importance: High

@enricosadahttps://github.com/enricosada I agree it is annoying, but we cannot keep Jenkins running tests in a PR, as the job definitions are regenerated on the CI server, not pulled from every commit. I'm planning on doing work on the GAC items in the next period, so this should be resolved.

@mmitchehttps://github.com/mmitche any pointers on using the elevated-permissions machine for now?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/859#issuecomment-170117124.

[enricosada]

Job definition can be the same. Just change cigroovy to pass different arguments to appveyorbuild script

Inviato da Mobile

On Fri, Jan 8, 2016 at 12:36 PM -0800, "Omar Tawfik" notifications@github.com wrote:

@enricosada I agree it is annoying, but we cannot keep Jenkins running tests in a PR, as the job definitions are regenerated on the CI server, not pulled from every commit. I'm planning on doing work on the GAC items in the next period, so this should be resolved.

@mmitche any pointers on using the elevated-permissions machine for now?

---

Reply to this email directly or view it on GitHub:
#859 (comment)

[mmitche]

FYI We can bring up more machines eventually with elevated permissions eventually, but you’ll see reduced TP for now. The other projects generally do not use elevated permissions (mostly because there is a tendency for bad practices to creep in, especially xplat)

From: Omar Tawfik [mailto:notifications@github.com]
Sent: Friday, January 8, 2016 12:36 PM
To: Microsoft/visualfsharp visualfsharp@noreply.github.com
Cc: Matt Mitchell mmitche@microsoft.com
Subject: Re: [visualfsharp] Jenkins build fails (#859)
Importance: High

@enricosadahttps://github.com/enricosada I agree it is annoying, but we cannot keep Jenkins running tests in a PR, as the job definitions are regenerated on the CI server, not pulled from every commit. I'm planning on doing work on the GAC items in the next period, so this should be resolved.

@mmitchehttps://github.com/mmitche any pointers on using the elevated-permissions machine for now?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/859#issuecomment-170117124.

[OmarTawfik]

@mmitche I agree. I've planned work in this area to remove this requirement in the future.
@enricosada if we change that to build only, we won't see the issues, and we would be blocked on effort to fix them incrementally. For #860, how about:

1. changing machine to 'windows-elevated'.
2. renaming jobs from "Jenkins {configuration}" to "[WIP] Jenkins {configuration}"?
   This should clear up the confusion a little bit until all other issues are resolved.

[enricosada]

@otawfik-ms ah my bad, the jenkins doesnt read the netci.groovy each run? like appveyor.ylm?

[mmitche]

@enricosada No, since it's generating much more than just the PR jobs.

[enricosada]

well, it's the same, we can use a jenkins-build.cmd so the build job definition is the same, and we can change behaviour inside each commit ( or branch / pr )

the jenkins-build.cmd call the appveyor-build.cmd, so it's the same.

also one job maybe is enogh, we create multiple jobs because there is a 1h timeout in appveyout, but that's not the case in jenkins, so maybe we can have only one Windows Release job.
my feedback 😄

[enricosada]

ok like #860 now, a jenkins-build.cmd.
that's good because we can tweak build, without need to change jenkins job definition

[ghost]

@enricosada I like the idea. We can go ahead with that.

[enricosada]

ok, updated #860

making the same as https://github.com/dotnet/cli/blob/master/netci.groovy only using jenkins-build.cmd Release
i added Debug too because it's important, if the build job definition works, it's possible to update appveyor-build to support a debug argument ( an we need only to change jenkins-build.cmd, not the job def )

[enricosada]

@oamsath but i cannot regen build defs, but the pr is ok i think

[enricosada]

work in progress to reenable tests from @TyOverby in #888