Enumerate live logger test matrix

[marcpopMSFT]

Potential scenario test matrix:

Examples

• Single project quick build
• Single project quick build with error/warningSingle project slow build
• Single project slow build with error/warning
• Medium number of projects
• Medium number of projects with error/warning (the error should be visible throughout the build)
• Medium number of projects with lots of the same error (can/should these be collapsed together)
• Medium number of project with lots of different errors
• Many projects with parallel builds
• Build various .net repos successfully
• Build various .net repos with failure(s)
• etc.

[marcpopMSFT]

This is for manual testing and analysis. Think about test cases we could hand off to our vendors. Additionally, there may be some testing we can do in an automated fashion but since this is a Ux heavy feature, that may be more difficult. Suggestions welcome.

[marcpopMSFT]

CC @baronfel

[rainersigwald]

Most of the testing so far has been "build the MSBuild repo":

.\build
.\scripts\Deploy-MSBuild.ps1 -destination .\.dotnet\sdk\7.0.101\
.\build -msbuildengine dotnet -ll

[AR-May]

I think it is too early now to write test matrix. It does not seem to me that set of features already set for live logger.

[rokonec]

Bellow test cases are designed to give us good enough coverage.

Test Case	Single Console	Multi project with tests	Orchard	non SDK .NET	non SDK C++
rebuild /m:1	✔️	✔️	✔️	✔️	✔️
rebuild /m	✔️	✔️	✔️	✔️	✔️
rebuild w/ cancel	✔️		✔️
restore	✔️	✔️	✔️
interactive restore	✔️	✔️	✔️
rebuild w/ errors	✔️	✔️	✔️	✔️	✔️
rebuild w/ lots of errors		✔️	✔️
restore w/ errors		✔️
dotnet test		✔️	✔️
test w/ MSBuild server opt in			✔️

I have added dotnet test intuitively, as I know it uses Console colors and Console.Write for test progress reporting.

As for "Multi project with tests" I would recommend sln with one console, three class libraries projects and one test xunit project.

By interactive restore I mean a restore which DeviceFlow prompt:

Here are some simple steps how to prepare simple console app:

• Running installcredprovider per these instructions
• If it was already installed we might need to delete cached credentials from: $env:UserProfile\AppData\Local\MicrosoftCredentialProvider
• If it was built before delete .\bin and .\obj folder

> dotnet new nugetconfig
> dotnet nuget add source https://pkgs.dev.azure.com/dnceng/internal/_packaging/7.0.100-rc.1.22429.3-shipping/nuget/v3/index.json
> dotnet new console
> dotnet build -r win-x64 --interactive --restore

[rokonec]

@ladipro @rainersigwald There might be other non-intuitive risky test cases which I have missed - please feel free to recommend.

[ladipro]

I understand these are manual tests and passing is defined as "looks reasonable and doesn't crash". Please correct me if I'm wrong.

Since the logger makes assumptions about the shape of the build in terms of target names, output messages, and maybe more, it would be interesting to also test custom projects not based on the .NET SDK. And because we have decided to also support the logger in MSBuild.exe, I wonder if we should test C++, non-SDK-style, and whatever else is commonly built on the VS command line.

[rokonec]

we should test C++, non-SDK-style, and whatever else is commonly built on the VS command line

Good point. In order to have test matrix manageable, I have added only non-SDK and C++ for some test cases.

[ladipro]

Filed #8724 to define a test plan using the data in this issue.