vstest.console.exe failed with return code: 4294967295

[marcwmiller]

I'm getting the following error intermittently (approx 10-20% of CI build/tests) with the Visual Studio test runner (version 2.0.30):

Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe failed with return code: 4294967295

My solution is VS2017 with .Net Core and Xunit test projects. I'm using the Hosted VS2017 agent.

Not only is it intermittent, but when it does fail, it is not in the same test project. Never are there failed tests.

What is return code 4294967295?

6_Test Assemblies .Tests.dll.txt

[janpieterz]

I also experience this lately, nothing to note with debug logging on, it just quits.
I also have it with some msbuild tasks (same idea, .net core in solution, xunit, hosted vs2017 agent).

Started happening since Tuesday

[daniefvh]

Same here. Just randomly started happening. It is a major blocker/annoyance for us.

[acesiddhu]

@AbhitejJohn can you check

[AbhitejJohn]

Tagging @smadala . Can someone share the diagnostic log too please(with System.Debug "true")?

[janpieterz]

Two attached, both debug set to true, same project, failed at different location with no 'reason'. Couldn't find anything significant.

logs_failed_msbuild.zip
logs_failed_vstest.zip

[smadala]

@janpieterz, @daniefvh To run .net core unit tests choose vstest.console.exe from "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe".
in vstest task (Execution options -> Specific location). In "Other console options" pass /diag:log.txt for diag information. please attach log.txt and log.*.host.txt.

@marcwmiller Please enable /diag:log.txt and attach log.txt and log.*.host.txt.

[janpieterz]

Seems for vstest to have something to do with code coverage. Where do I get those log.txt files from?
Also any thoughts about the msbuild one?

logs_639.zip

[smadala]

Seems for vstest to have something to do with code coverage.

/EnableCodeCoverage yet to support in .net core. Tracking here.

Pass /Framework:FrameworkCore10 arg in "Other console options" for running .net core tests.
log.txt is relative path from current working directory, you can pass absolute path too.

No vstest task or tests running in msbuild one. It was failing in Build solution, @acesiddhu Can you please tag Build solution task folks?

[nigurr]

We are adding respective folks to investigate into this issue. Seems like both msbuild and vstest are failing with same error code.

[marcwmiller]

@smadala - I have enabled /diag:log.txt as requested and will attach log files after next failure.

[marcwmiller]

@smadala - As luck would have it (I guess) I got the error first time running CI build after enabling diagnostic logs. Please find log files attached.
6_Test Assemblies .Tests.dll.txt
log.host.17-05-08_12-07-31_43223_4.txt
log.host.17-05-08_12-07-55_11017_4.txt
log.host.17-05-08_12-07-57_53449_4.txt
log.host.17-05-08_12-08-02_33688_4.txt
log.host.17-05-08_12-08-07_38614_9.txt

[smadala]

@marcwmiller

DefaultEngineInvoker: ParentProcess '1244' Exited in log.host.17-05-08_12-08-07_38614_9.txt

Looks vstest.console.exe exiting abnormally, Can please share log.txt which will be in current directory of vs test task command runs.
vstest.console.exe full dump by following instructions here.

[marcwmiller]

@smadala - I think I've correctly updated the Copy Files task to capture the log.txt file (we'll have to wait for another failure).

But I'm not sure about your last request for logs - I'm using the Hosted VS2017 agent and cannot perform instructions provided in the link. Please advise.

[naile]

This seems widespread, we have this error in approx. every 3rd build or so for several projects. Sometimes consecutive test runs have this, then it works for a couple of builds without any apparent change on our side. This happens only in VSO i haven't been able to repro locally.

Solution is VS 2017 .Net 4.6, Nunit 3.7, Hosted VS2017 agent

[smadala]

@marcwmiller are you able to get log.txt file?
@nigurr Is there a way we can get dmp from Hosted server?

[marcwmiller]

@smadala - just got error moments ago. Please find attached log files.
6_Test Assemblies .Tests.dll.txt
log.host.17-05-08_18-17-32_22127_4.txt
log.host.17-05-08_18-17-58_32720_4.txt
log.host.17-05-08_18-18-00_77082_4.txt
log.host.17-05-08_18-18-06_21002_4.txt
log.host.17-05-08_18-18-10_73618_4.txt
log.host.17-05-08_18-18-16_22168_4.txt
log.txt

[daniefvh]

Just a note that we are not running .NET core. I will try getting logs for you.

[nigurr]

@marcwmiller can you please send your VSTS account and build details to devops_tools at microsoft dot com
I will be investigating into this asap.

[nvankesteren]

FYI, we have the same problem with .net 4.6, VS2017, 2017 host, MsTest v2. And it happens to the build step as well, about as frequent.

[nigurr]

Hi,

We have created a Live Site Incident for the same and actively working on this.

[marcwmiller]

@nigurr - I am unable to email the information requested. My email is returned with the following error:

The groups only accept messages from people in their organization or on their allowed senders list, and your email address isn't on the lists

[nigurr]

@marcwmiller replied to you

[chrispat]

does everyone on this thread have .NET Core projects in their solution? Also can you all post your account URLs?

[marcwmiller]

@chrisrpatterson - My solution contains .NET Core projects. I have shared account URL with @nigurr and can email to you if you'd like.

[chrispat]

@marcwmiller can you simply ping your account and tell me what it resolves to.

[janpieterz]

Jup, contains .NET core, account url is intreba.visualstudio.com, one of affected projects Arke.
@nigurr has access to this already.

[marcwmiller]

@chrisrpatterson tfsprodcus1.app.visualstudio.com [104.43.203.175]

[chrispat]

Another question, what about TypeScript?

[chrispat]

@janpieterz while I am working through debugging this issue I am going to temporarily increase the number of hosted agents in your account so I don't block your normal work

[janpieterz]

@chrisrpatterson I see, interesting. I'll wait for you to finish, only need to build correctly once for this security advisory.

[chrispat]

@janpieterz you have 5 agents right now so you should be able to build.

[janpieterz]

Jup, I noticed when both ran at the same time. Cheers!

[nvankesteren]

We don't have any .NET Core / TypeScript stuff.

We went back to the non-2017 host and all is well. We can't open it up for experimenting with builds since we have external dependencies (database for end2end tests).

Like I said in a previous post, we see builds failing on either the build step or the test step (not both).

[chrispat]

@janpieterz thank you for your patience. We are working with the MSBuild team to figure out what additional diagnostics we can gather. It seems the failure rate is about 20% and the MSBuild failure happens at a different place in the project every time.

[janpieterz]

@chrisrpatterson no worries, happy to help! If you want to speed up your build test, you could disable webpack and npm, they shouldn't bother the msbuild and test!

If you'd like me to add any other MS employee just send me an email or ping me here, besides that I'm enjoying the notifications for the builds in our Slack 😄

[chrispat]

@janpieterz we have a repro from a basic solution https://github.com/sgrebnov/Windows-appsample-coloringbook now. You can remove me from your account if you want. For a temporary work around if you add a second vsbuild task right after the first one and set the first to continue on error it seems to complete successfully 100% of the time.

[janpieterz]

@chrisrpatterson Great to hear, just removed you. In case you need to get back in just let me know, otherwise good luck hinting, and cheers for that temp workaround!

[janpieterz]

What do I do with those agents btw?

[chrispat]

@janpieterz the extra hosted agents should disappear. You can also delete the agent in the default pool.

[cob999]

we are also seeing about 50% of builds fail on running tests with error

Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 4294967295

similar to @nvankesteren we are using the Hosted VS2017 agent, due to this we also needed to change the build to use the Version 2.* preview version of the Visual Studio Test step

The current situation is intolerable, but not sure what to do, is the error we are seeing the same issue? is this a known issue that is being fixed or do we back out and stop using the Hosted VS2017 agent? we started using the VS2017 agent because the default one started giving us c# compilation errors randomly due to new c# language features.

[bryanmacfarlane]

We are actively working very hard on it. We apologize for the inconvenience this is causing. What we know is the process does not crash - both msbuild and vstest simply exits with -1 or maxint. It only happens on the first run on a fresh OS (that is the hosted build scenario since every build is a new machine reimage). For example, you can put a second task of build or test in the same job with the first being continue on error and it will always succeed.

The workaround is to use a private agent (your own VM / machine).

[bryanmacfarlane]

A new image which fixes/mitigates the issue should be rolling across scale units throughout the day. Chime in throughout the weekend and early in the week if it's addressed for you. We will be keeping an eye on it.

[marcwmiller]

@bryanmacfarlane - sweet. Thanks to all involved with issue resolution.

[cob999]

got another failure just now
Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 4294967295

[nigurr]

@cob999, can you please send your VSTS account and build details to devops_tools at microsoft dot com
and send your build log?

[cob999]

@nigurr I've sent that on, got a bounce message saying Couldn't deliver to the following groups: but i'm hoping the original email got to you

[marcwmiller]

@nigurr - I've received the error twice today :-(

Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe failed with return code: 4294967295
Vstest exited with return code: 1.

[bryanmacfarlane]

we're copying a new image with gflags to figure out what's wacking the process (it's not crashing, it's not exiting). the previous mitigation appears to reduce the likelyhood but still hapenning ... we're on it.

[nigurr]

A new image is rolled out with additional logging enabled to determine more information. When msbuild.exe/vstest.console.exe exits, info will be written to the application event log with the exit code and if another process killed it, the name of the process that killed msbuild.exe/vstest.console.exe.

Please add the below inline powershell to grab the info from the application event log once. Please make sure you select "Always Run" in the task so that any previous errors in the build will still continue to execute this and dumps the logs.

Get-EventLog -LogName Application -After ([datetime]::now.addhours(-1)) -Source 'Process Exit Monitor' | % { "EventID: $($_.EventID)" ; "TimeGenerated: $($_.TimeGenerated)" ; "Message: $($_.Message)" ; "" }

[nvankesteren]

Can MS repro this without outside projects? My team no longer has the resources to keep testing this and have stepped back a version. I'm watching this space to see if the issue is resolved.

[bryanmacfarlane]

We finally got a repro which captured the offending process from the event log line above. We know exactly what is killing msbuild/vstest now. We're following up right now with an image update. It takes awhile to rollout but good news.

This one was especially tough because the size, length and volatility of our tool would protect it from repro'ing. Sorry for the inconvenience.

[marcwmiller]

@bryanmacfarlane - when do you anticipate the roll out to be complete? I'm still receiving error:

Ran for 3.7 minutes (Hosted VS2017), completed at Fri 05/26/2017 05:43 PM
Request Summary
Request 309 Marc Miller Completed
Summary
| Build
2 error(s), 3 warning(s)
Build - 2 error(s), 3 warning(s)
Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe failed with return code: 4294967295
Vstest exited with return code: 1.

[ericsciple]

@thechrisjohnson should this have deployed everywhere by now?

[ericsciple]

@marcwmiller please send ersciple the name of your account (at microsoft com).

[marcwmiller]

@ericsciple - done.