data collection support in vstest

[sbaid]

Description

data collectors are an extensibility point for the test platform. What is the CLI experience for the data collection support?

vstest.console /collect:coverage;filename=foo.coverage
vstest.console /collect:events;filename=bar.log

I am not sure whether integrating opencover as a data collector make sense. I see that it is currently has a command line tool with rich set of arguments.

Also what will be the way to pick the data collector? like logger.

[sbaid]

@pvlakshm @codito @harshjain2 @mayankbansal018 @Faizan2304

[harshjain2]

Also what will be the way to pick the data collector? like logger.
Earlier we had a discussion on this and there was an consensus on having a new tag "ExtensionsPaths" in RunSettings, similar to "TestAdaptersPaths" where we can put all the extensions, be it logger or a data collector.

[harshjain2]

What is the CLI experience for the data collection support?
IMHO, we should not support any first class argument for data collectors, as each data collector can have their own set of parameters that can be declared in runsettings.

For code coverage, there can be an exception to support first class argument since we can use default parameters in case user hasn't provided explicit parameters in runsettings. That's how /EnableCodeCoverage was supported in TPV1.

[codito]

Some open questions we need to decide on.

Acquisition of data collectors

Choices:

1. Nuget packages.
   • Similar to test adapters, all *.datacollectors.dll available in nuget dependency of a test project are marked as data collector extensions.
   • This means datacollectors will load/unload for every test project (at the cost of perf). Can the datacollectors merge the reports etc. if multiple projects are passed for a run?
   • Ease of managing these extensions = add/remove nuget packages.
   • Works for any IDE/Editor/CLI since the configuration is with the project itself.
2. /extensions:<path> command line argument
   • *.datacollectors.dll are picked from user provided path. They are loaded for the entire run.
   • It is difficult to find the path of such packages in netcore (since they are not in dependency of any project)
   • For IDE/Editors this argument needs to be available in runsettings.
3. Exact assembly specification provided in runsettings.
   • Run settings provides the fullyqualified assembly name of a datacollector. These are loaded with default CLR assembly resolution logic, and a special assembly resolver to load inbox collectors
   • Datacollectors can ship as nuget packages which copy dlls as content to test project output directory.
   • For IDE/Editors this needs to be provided along with TestRunRequest.

Enable/disable of data collectors

Choices:

1. All data collectors specified in runsettings are enabled by default. User needs to modify runsettings to disable.
2. Acquisition of datacollectors is a separate. The data collectors to be enabled need to be passed in as /collect:<friendly name> in command line arguments. Here IDE/Editors need an extensibility in TestRunRequest.

Configuring a data collector

A data collector can be configured with runsettings or with CLI arguments passed in with -- codecoverage.filename=foo.log

Authoring own datacollector

1. User can author any .net based data collector. They will load with default host.
2. The data collector protocol should be open/published as part of SDK.
3. Any IDE/Editor/Extension author can plugin their own datacollector host. vstest.console will launch the host and communicate with json (similar to testhost extensibility).
   Is 2 and 3 part of current scope?

[pvlakshm]

Acquisition – Choice 3
Enable/disable of data collectors – Choice 1
Configuring a data collector – as called out above
Authoring own datacollector – as called out above
CodeCoverage – support /EnableCodeCoverage as a special commandline switch (similar to TPV1)

[sbaid]

we need a better way to enable extensions (data collectors, loggers) through first class CLI switch without using runsettings.

[sbaid]

The spec is here - https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md

[aKabirKhan]

Any update on this? we need test coverage to work in .net core application, stranded a bit.

[codito]

@aKabirKhan Code coverage for .net core is available on VS 2017 15.3 Preview 1 (download). Instructions to try out are at https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md#working-with-code-coverage. Please try it out, let us know if you have any feedback.

[codito]

Closing this issue since data collector support is available in test platform now. Some work is pending for code coverage support, it is tracked in #579

[aKabirKhan]

Thank you @codito. I will give it a go.

[xperiandri]

What could be an issue if analyze report has <skipped_module reason="no_symbols" />
Why symbols can be ignored?
I checked, symbols exist.

[xperiandri]

Portable PDBs not yet supported until 15.4 according to https://developercommunity.visualstudio.com/comments/84459/view.html