TestContext.Progress shows up as Warning in TFS/Azure Dev Ops

[mharwig]

Version information

• NUnit3TestAdapter version: 4.3.1.0
• NUnit 3.13.3
• Visual Studio edition: Microsoft Visual Studio Professional 2022 (64-bit) - Current
  Version 17.4.2
• What .net platform and version is being targeted: .Net 7.0
• TFS/VSTS: Azure DevOps Server Version Dev18.M170.8 on premise
  • Task: VSTest@2 (https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/vstest-v2?view=azure-pipelines&viewFallbackFrom=azure-devops)

Issue

TestContext.Progress written during a test shows up in TFS/Azure DevOps as warning when running a test via a VSTest task with NUnit Adapter 4.3.1.0. ConsoleOut is enabled via runsettings.

Code used to write progress in test:
TestContext.Progress.WriteLine("Message via TestContext.Progress");
Expected result white (informational) log line in TFS. Instead the message is shows as a warning (orange) warning in TFS:

Possible cause

My theory regarding the cause of the issue based on analysis of the code (did not run/debug the code):
The text written via TestContext.Progress.WriteLine results in a TestOutput event in EventListenerTextWriter:

private bool TrySendToListener(string text) { context.Listener.TestOutput(new TestOutput(text, _streamName, context.CurrentTest?.Id, context.CurrentTest?.FullName));

The TestOutput event is in handled in nunit3-vs-testadapter in class NUnitEventListener OnTestEvent:

public void OnTestEvent(string report) { . . . case NUnitTestEventHeader.EventType.TestOutput: TestOutput(new NUnitTestEventTestOutput(node)); break;

The handled TestOutput is always forwarded as warning in NUnitEventListener TestOutput:

public void TestOutput(INUnitTestEventTestOutput outputNodeEvent) { . . . recorder.SendMessage(TestMessageLevel.Warning, text); }

Possible solution:
recorder.SendMessage(outputNodeEvent.IsProgressStream ? TestMessageLevel.Informational : TestMessageLevel.Warning, text);
or
recorder.SendMessage(outputNodeEvent.IsErrorStream ? TestMessageLevel.Warning : TestMessageLevel.Informational, text);
Further more one could argue that if settings.ConsoleOut == 1 the message level should also be TestMessageLevel.Informational, similar to the message sent in NUnitEventListener TestFinished.

Apologies if the issue turns out to be caused by something else entirely (e.g. configuration on my side).

[mharwig]

I just noticed I only have this issue when running with batched tests, so vstest task settings
batchingBasedOnAgentsOption: customBatchSize
customBatchSizeValue: 100

[OsirisTerje]

@mharwig Those settings don't do anything wrt the adapter. Can you provide a repro for this? Using a public azure pipeline and git repo would do.

[mharwig]

@OsirisTerje I realize that those settings are not related to the adapter. However using customBatchSizes has the side effect that TFS runs the tests via the translation layer (see comment here: microsoft/azure-pipelines-tasks#10709 (comment)), as opposed to calling vstest.console.exe directly. Apparently the variant ran via the api handles the warning messages differently. I will see if I can find the time to create a reproducer.

[OsirisTerje]

Your idea of using the ConsoleOut==1 could be a consistent solution. Alternatively, introduce explicit settings for how to handle Progress and Error.

[mharwig]

@OsirisTerje See https://martinharwig.visualstudio.com/NUnitVsTestAdapterBug/_build/results?buildId=9&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9&t=7bfb7e0f-549d-5929-c603-6e8046613b9e for a reproducer.

[mharwig]

Your idea of using the ConsoleOut==1 could be a consistent solution. Alternatively, introduce explicit settings for how to handle Progress and Error.

Explicit settings would also be fine with me. However the suggested solution using IsErrorStream to output warning and informational otherwise would also work and is consistent with output handled by TestConverter.FillResultFromOutputNodes. Explicit settings would of course be more flexible.

[OsirisTerje]

Thanks for the repro, I see your point there. You could just raise a Pull Request with your suggested change. I can't see any reason for not approving that.

[mharwig]

Raised a pull request, please note that Windows task of 'NUnit 3 VSTest Adapter CI ' currently fails for all branches. I'm guessing the 'Install .NET Core 2.1 runtime' step should be changed to 'Install .NET Core 3.1 runtime', which is something I cannot do.

[OsirisTerje]

Thanks!

PS: Yes, aware of those fails. Will be fixed. :-) But the Cake build is green, and that is the important one.

[mharwig]

Thanks for the quick responses, btw I tested the resulting changes on the reproducer and it had the desired effect

[OsirisTerje]

Awesome!

[OsirisTerje]

@mharwig Version 4.4.0-beta.1 is out now https://www.nuget.org/packages/NUnit3TestAdapter/4.4.0-beta.1 . It includes this issue and support for -net 8. Appreciate if you have the time to verify it works as it should.

[mharwig]

@OsirisTerje Hi, Thanks for the update.

I just updated to Version 4.4.0-beta.1 in the reproducer, and it works as intended.

Run with version 4.3.1:
https://martinharwig.visualstudio.com/NUnitVsTestAdapterBug/_build/results?buildId=14&view=results

Run with Version 4.4.0-beta.1 (no more warning with progress message):
https://martinharwig.visualstudio.com/NUnitVsTestAdapterBug/_build/results?buildId=15&view=results