Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nunit3-console with altcover 0% coverage #47

Closed
39547361 opened this issue Jan 4, 2019 · 13 comments
Closed

nunit3-console with altcover 0% coverage #47

39547361 opened this issue Jan 4, 2019 · 13 comments
Labels
documentation

Comments

@39547361
Copy link

39547361 commented Jan 4, 2019

For some strange reason i get 0% coverage with altocover (monocov reports 90%). Am i missing something? Any hints would be greatly appreciated.

altcover.5.0.663
MacOSX 17.7.0.0
CLR Version: 4.0.30319.42000

Step 1: create pcl and nunit test dll

nuget restore
/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild TESTReport.sln /t:TESTReportPCL /p:Configuration="Debug" /p:BuildProjectReferences=false /m  /verbosity:m
/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild TESTReport.sln /t:NUnit_TESTReport /p:BuildProjectReferences=false /m  /verbosity:m

Step 2

cd NUnit.TESTReport/bin/Debug/

Step 3

mono ../../../packages/altcover.5.0.663/tools/net45/AltCover.exe --opencover --linecover  --assemblyFilter=.+.TESTReport+ --assemblyFilter=TESTReport+ --assemblyExcludeFilter=nunit+ --assemblyExcludeFilter=.+\.Tests --assemblyExcludeFilter=NUnit+ --assemblyExcludeFilter=AltCover.+ --assemblyExcludeFilter=Xamarin+ --assemblyExcludeFilter=SQL+ --assemblyExcludeFilter=System+ --assemblyExcludeFilter=Plugin+ --assemblyExcludeFilter=PCLExt+ --assemblyExcludeFilter=FFImage+ --assemblyExcludeFilter=Newtonsoft+  --outputDirectory=__Instrumented

Output:

Instrumenting files from /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug
Writing files to /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/TestReportPCL.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/FFImageLoading.Forms.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/FFImageLoading.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/NUnit.TestReport.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Newtonsoft.Json.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Plugin.Connectivity.Abstractions.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Plugin.Connectivity.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/SQLite-net.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/SQLiteNetExtensions.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Xamarin.Forms.Core.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/Xamarin.Forms.Xaml.dll
   => /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/nunit.framework.dll

Coverage Report: /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/coverage.xml


    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/nunit.framework.dll
                <=  nunit.framework, Version=3.11.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Xamarin.Forms.Core.dll
                <=  Xamarin.Forms.Core, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/SQLite-net.dll
                <=  SQLite-net, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Plugin.Connectivity.Abstractions.dll
                <=  Plugin.Connectivity.Abstractions, Version=3.2.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Newtonsoft.Json.dll
                <=  Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/FFImageLoading.dll
                <=  FFImageLoading, Version=2.4.4.859, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Xamarin.Forms.Xaml.dll
                <=  Xamarin.Forms.Xaml, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/SQLiteNetExtensions.dll
                <=  SQLiteNetExtensions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/Plugin.Connectivity.dll
                <=  Plugin.Connectivity, Version=3.2.0.0, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/FFImageLoading.Forms.dll
                <=  FFImageLoading.Forms, Version=2.4.4.859, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/TestReportPCL.dll
                <=  TestReportPCL, Version=1.0.6941.28281, Culture=neutral, PublicKeyToken=null
    /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug/__Instrumented/NUnit.TestReport.dll
                <=  NUnit.TestReport, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null

Generated files:

AltCover.Recorder.g.dll
TestReportPCL.dll
TestReportPCL.dll.config
TestReportPCL.pdb
CarouselView.FormsPlugin.Abstractions.dll
coverage.xml
FFImageLoading.dll
FFImageLoading.Forms.dll
FFImageLoading.Forms.pdb
FFImageLoading.Forms.xml
FFImageLoading.pdb
FFImageLoading.Platform.dll
FFImageLoading.Platform.xml
FFImageLoading.Transformations.dll
FFImageLoading.Transformations.xml
FFImageLoading.xml
I18NPortable.dll
libe_sqlite3.dylib
Newtonsoft.Json.dll
Newtonsoft.Json.pdb
Newtonsoft.Json.xml
NUnit.TestReport.dll
NUnit.TestReport.dll.config
NUnit.TestReport.pdb
nunit.framework.dll
nunit.framework.pdb
nunit.framework.xml
PCLExt.FileStorage.Abstractions.dll
PCLExt.FileStorage.Abstractions.xml
PCLExt.FileStorage.dll
PCLExt.FileStorage.xml
Plugin.Connectivity.Abstractions.dll
Plugin.Connectivity.Abstractions.pdb
Plugin.Connectivity.Abstractions.xml
Plugin.Connectivity.dll
Plugin.Connectivity.pdb
Plugin.Connectivity.xml
Plugin.Media.dll
Plugin.Media.xml
Plugin.Permissions.dll
Plugin.Permissions.xml
SignaturePad.Forms.dll
SignaturePad.Forms.xml
SQLite-net.dll
SQLite-net.pdb
SQLite-net.xml
SQLiteNetExtensions.dll
SQLiteNetExtensions.pdb
SQLitePCLRaw.batteries_green.dll
SQLitePCLRaw.batteries_v2.dll
SQLitePCLRaw.core.dll
SQLitePCLRaw.provider.e_sqlite3.dll
System.Data.Common.dll
System.Diagnostics.StackTrace.dll
System.Diagnostics.Tracing.dll
System.Globalization.Extensions.dll
System.IO.Compression.dll
System.IO.dll
System.Linq.dll
System.Linq.Expressions.dll
System.Net.Http.dll
System.Net.Sockets.dll
System.Reflection.dll
System.Runtime.dll
System.Runtime.Extensions.dll
System.Runtime.InteropServices.dll
System.Runtime.InteropServices.RuntimeInformation.dll
System.Runtime.Serialization.Primitives.dll
System.Security.Cryptography.Algorithms.dll
System.Security.Cryptography.Encoding.dll
System.Security.Cryptography.Primitives.dll
System.Security.Cryptography.X509Certificates.dll
System.Security.SecureString.dll
System.Text.RegularExpressions.dll
System.Threading.Overlapped.dll
System.Xml.ReaderWriter.dll
System.Xml.XPath.XDocument.dll
Xamarin.Forms.Core.dll
Xamarin.Forms.Core.pdb
Xamarin.Forms.Core.xml
Xamarin.Forms.Platform.dll
Xamarin.Forms.Xaml.dll
Xamarin.Forms.Xaml.pdb
Xamarin.Forms.Xaml.xml

coverage xml looks fine

Step 4:

mono ../../../packages/altcover.5.0.663/tools/net45/AltCover.exe Runner --recorderDirectory __Instrumented  --executable ../../../packages/NUnit.ConsoleRunner.3.9.0/tools/nunit3-console.exe   -- ../../../NUnit.TESTReport/bin/Debug/NUnit.TESTReport.dll

Output:

Beginning run...
Command line : '../../../packages/NUnit.ConsoleRunner.3.9.0/tools/nunit3-console.exe ../../../NUnit.TestReport/bin/Debug/NUnit.TestReport.dll'
NUnit Console Runner 3.9.0
Copyright (c) 2018 Charlie Poole, Rob Prouse
Runtime Environment
   OS Version: MacOSX 17.7.0.0 
  CLR Version: 4.0.30319.42000
Test Files
    ../../../NUnit.TestReport/bin/Debug/NUnit.TestReport.dll
Run Settings
    DisposeRunners: True
    WorkDirectory: /Users/jenkins/workspace/TestClientForEditors/NUnit.TestReport/bin/Debug
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.7.1
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    NumberOfTestWorkers: 8
Test Run Summary
  Overall result: Passed
  Test Count: 99, Passed: 99, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2019-01-03 08:34:47Z
    End time: 2019-01-03 08:34:56Z
    Duration: 9.515 seconds
Results (nunit3) saved as TestResult.xml
Getting results...
A total of 0 visits recorded
Coverage statistics flushing took 0.13 seconds
Visited Classes 0 of 39 (0)
Visited Methods 0 of 71 (0)
Visited Points 0 of 649 (0)
Visited Branches 0 of 71 (0)
==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 0 of 46 (0)
Alternative Visited Methods 0 of 119 (0)
@SteveGilham
Copy link
Owner

Looks like the coverage results aren't being properly flushed to disk on process termination; the same underlying issue as #42

@SteveGilham SteveGilham added the bug label Jan 9, 2019
@39547361
Copy link
Author

release/v4.0.659 (that fixed #42) does not fix this issue

@SteveGilham
Copy link
Owner

Actually, issue #42 was re-opened a couple of weeks before this issue was raised. Getting a repro on that is next on my list when I get a chance to resume work on this project.

@39547361
Copy link
Author

Sorry for misleading comment. What i meant to say was:
This issue still present with an older altcover release/v4.0.659
it might be relevant in so far that issue #42 is not occurring with altcover release/v4.0.659

@SteveGilham
Copy link
Owner

As the only change near the relevant code between 4.0.659 and 5.0.663 was a change in the static linkage of the F# core assembly, the original "fix" may have been more a "working by coincidence".

Given that the whole system relies on any outstanding data being flushed during the process shutdown handler, I can see a possibility where very small or very large programs-under-test work, but moderately sized ones fail with zero coverage, so I need to test that hypothesis.

@SteveGilham
Copy link
Owner

I've not been able to get any reliable repro, but I have done some tuning on the recording side in release 5.0.665, which may ameliorate this problem.

@SteveGilham SteveGilham added the cannot repro Need a simple case of the issue to work from label Feb 2, 2019
@39547361
Copy link
Author

39547361 commented Feb 8, 2019

Thank you so far for your effort. Unfortunately 5.0.665 does not fix this particular issue.
I created a private sample project and sent you an invite in case you are interested.

@SteveGilham
Copy link
Owner

I was afraid that I wouldn't have fixed it; I have a very intermittent case which showed up for the first time in a while after I'd made the release.

I'm not seeing your invite anywhere obvious from my side, though.

@SteveGilham
Copy link
Owner

The invite appeared a mailbox sub-folder, rather than anywhere on my GitHub profile. I hadn't been expecting that.

@SteveGilham SteveGilham removed the cannot repro Need a simple case of the issue to work from label Feb 8, 2019
@SteveGilham
Copy link
Owner

Thank you for that. It reproduces on Windows.

@SteveGilham
Copy link
Owner

SteveGilham commented Feb 8, 2019 •
edited
Loading

The first thing I noticed in looking at the coverage.xml file was that every assembly was marked skippedDueTo="Filter"; the second was that the module under test, AltcoverTestPCL wasn't even present there.

I proceeded to investigate by the following steps

  1. recursively delete NUnit.AltcoverTest/bin/Debug/
  2. rebuild NUnit.AltcoverTest.csproj
  3. cd NUnit.AltcoverTest/bin/Debug/
  4. run AltCover.exe --opencover --linecover --assemblyExcludeFilter="^((?!AltcoverTestPCL).)*$" --outputDirectory=__Instrumented using a regex to simply exclude every assembly not containing AltcoverTestPCL in its name
  5. as this is Windows, manually copy the x86 and x64 folders to __Instrumented (a limitation; I could have used --inplace in step 4 instead); other platforms will just place libe_sqlite3.so in the Debug folder where it will get copied as expected
  6. run AltCover.exe Runner --recorderDirectory __Instrumented --executable ../../../packages/NUnit.ConsoleRunner.3.9.0/tools/nunit3-console.exe --./__Instrumented/NUnit.AltcoverTest.dll to point NUnit at the instrumented test system in __Instrumented.
  7. See the following
11,931 visits recorded in 00:00:00.2904547 (41,077 visits/sec)
A total of 11,931 visits recorded
Coverage statistics flushing took 0.38 seconds
Visited Classes 43 of 43 (100)
Visited Methods 83 of 84 (98.81)
Visited Points 543 of 582 (93.3)
Visited Branches 0 of 84 (0)
==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 43 of 54 (79.63)
Alternative Visited Methods 83 of 132 (62.88)

@SteveGilham SteveGilham added documentation ready to close Believed addressed, waiting to hear to the contrary and removed bug labels Feb 8, 2019
@SteveGilham SteveGilham removed the ready to close Believed addressed, waiting to hear to the contrary label Feb 16, 2019
@39547361
Copy link
Author

Thank you very much for resolving my misconfiguration. Have a great day sir!

@SteveGilham
Copy link
Owner

FWIW, the issue in point 5 of my comment of the 8th Feb 2019 is belatedly resolved -- the manual copy of the x86 and x64 folders is no longer needed, as this is done automatically since today's release of v8.2.825; at the time I was too busy focussing on the zero coverage issue to heed my own words about this being a limitation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation
Projects
Current release series (5.x, 6.x, 7.x, 8.x, ...)
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@SteveGilham @39547361