Migrate from xUnit to NUnit #79

leasunhy · 2017-04-28T05:43:33Z

This PR deprecates xUnit in favor of NUnit.

Update the test project template - .NET Framework.
Update the test project template - .NET Core.
Migrate existing tests.
Edit the root msbuild script files - test.csproj.
Edit the root msbuild script files - test_coreclr.csproj.
Create a little script to help test project migration.

…updated

msftclas · 2017-04-28T05:43:37Z

@leasunhy,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

yatli · 2017-04-28T06:33:08Z

Looking good, please do some testing with coreclr. Thanks!

yatli · 2017-04-28T06:52:20Z

tests/xunit2nunit.sed

+s@InlineData@TestCase@g
+s@Fact@Test@g
+s@using Xunit@using NUnit.Framework@g
+s@Assert.Equal@Assert.AreEqual@g


TODO add Assert.True -> Assert.That

leasunhy · 2017-04-30T14:04:38Z

Support for coreclr blocked due to nunit/nunit3-vs-adapter#313.
According to that PR, hopefully we will be able to support coreclr in a few days.

yatli · 2017-05-01T05:42:01Z

@leasunhy are you sure about this? IIRC nunit has always been working with dotnet test. The PR you referenced is for the visual studio adapter (vstest.exe, or to run the tests within VS).

…test projects

yatli · 2017-05-02T05:50:56Z

tests/nunit_tools/nunit_meta_runner/Program.cs

+        {
+            if (args.Length < 2)
+            {
+                Console.Error.WriteLine("Usage: runner <runner-path> <test-assembly> <result-dir> [nunitlite-arg ..]");


what is runner-path?

The testing process for one test assembly is:

Meta-runner scans the test assembly and get the list of tests.

Meta-runner starts a test-runner for each test in the test assembly.

(TODO) Merge the result XML files into one.

... where, the path of the test-runner is specified in runner-path.

yatli · 2017-05-02T05:51:34Z

tests/nunit_tools/nunit_meta_runner/Program.cs

+            var assemblyPath = Path.GetFullPath(args[2]);
+            var remainingOptions = args.Skip(3).ToArray();
+
+            // NOTE(leasunhy): this may fail silently if some of the dependencies are not preloaded!


no exceptions?

Yes, in our scenario, when NUnit is not preloaded, it will throw an exception, but when Trinity is not preloaded, it will succeed but no tests can be discovered.

yatli · 2017-05-02T05:52:46Z

tests/nunit_tools/nunit_meta_runner/Program.cs

+
+            foreach (var testName in allTestNames)
+            {
+                var process = CreateProcessForTest(runnerPath, assemblyPath, resultDirPath, remainingOptions, testName);


TODO we'd better add a timeout limit (for example 30mins). IIRC NUnitLite has this feature.

… run now

leasunhy · 2017-05-04T05:33:30Z

The --timeout option of NUnitLite under .NET Core behaves strangely. That option will block the test runner even after the tests should have finished.
When NUnitLite is used with .NET Framework, the problem is gone.

leasunhy · 2017-05-04T06:24:50Z

OK, actually NUnit on .NET Core does not support Timeout at all. Even TimeoutAtttribute does not exist.
The cause is rooted in .NET Core: it does not provide Thread.Abort()...
I guess we will have to do Timeout in the meta runner.
How do you think? @yatli

yatli · 2017-05-04T07:17:14Z

@leasunhy agreed. A bit of a hack, but see: https://github.com/dotnet/coreclr/blob/master/tests/src/baseservices/threading/regressions/threadex.cs

In the inner meta runner process, setup a timer event, which sends abort to record the test results.
Also, how about Thread.Interrupt()? Since we want the main thread of the test engine kept running, we should only inject an exception into it, not to abort it.

yatli · 2017-05-08T09:06:45Z

tests/nunit_tools/nunit_meta_runner/Program.cs

@@ -47,7 +51,7 @@ public static int Main(string[] args)

            foreach (var testName in allTestNames)
            {
-                var process = CreateProcessForTest(runnerPath, assemblyPath, resultDirPath, remainingOptions, testName);
+                var process = CreateProcessForTest(runnerPath, assemblyPath, resultDirPath, runnerOptions, testName);
                process.WaitForExit();
                Console.WriteLine(process.StandardOutput.ReadToEnd());
                // TODO(leasunhy): check the exit status of the process and regard the test as a failure


Just don't forget to add this feature :)

Sorry, which feature?

…ormally

…imeouts

leasunhy added 12 commits April 28, 2017 10:37

Test: move to nunit - test

efaeff3

Test: remove test project

20bf628

Test: add simple xunit-to-nunit converting scripts

a916a55

Test: update root packages.config for nunit

10dcb98

Test: updated template for nunit; project files for coreclr not updated

1458726

Test: update convert_to_unit script

32fe9a5

Test: convert existing test projects to nunit projects

f4d7af5

Test: update convert script to handle Assert.Equal

d4a4425

Test: update test.csproj to run nunit tests; test_coreclr.csproj not …

1c2213a

…updated

Test: fix Assert.Equal and Assert.InRange

12cd1ef

Test: update .gitignore to ignore nunit reports

f0f7574

Test: fix tsl1

ceb3e51

msftclas added the cla-already-signed label Apr 28, 2017

yatli reviewed Apr 28, 2017

View reviewed changes

leasunhy added 2 commits April 28, 2017 19:36

Test: replace Assert.True with Assert.That

7eec74c

Test: update msbuild script to make nunit spawn only one worker

303d796

yatli approved these changes May 1, 2017

View reviewed changes

leasunhy added 9 commits May 1, 2017 14:02

Test: coreclr support for NUnit wip

eecb1cd

Test: fix the permission mode for tests/convert_to_nunit.sh

dfa2910

Test: NUnit test runner init

ccc2f72

Test: update coreclr csproj template and accordingly update existing …

7ba6571

…test projects

Test: fix nunit test runner

56d9255

Test: update test_coreclr.csproj to make use of nunit_test_runner

bdaf801

Test: meta runner wip

69f815f

Test: meta runner milestone - process isolation done

052a24c

Test: meta runner - add argument result-dir

a5f417a

Test: update test_coreclr.csproj

4e28745

yatli reviewed May 2, 2017

View reviewed changes

leasunhy added 2 commits May 2, 2017 14:24

Test: refine test_coreclr.csproj - tests failing to build will not be…

9668787

… run now

Test: coreclr tests are run with a time limit of 30mins

7887fee

leasunhy force-pushed the test-nunit branch from 38e9f5d to 7887fee Compare May 2, 2017 07:23

leasunhy added 5 commits May 8, 2017 15:56

Test: omit timeout for now

30c14fb

Test: use new syntax for command line args for the meta runner

10a7080

Test: add timeout support in meta runner

d6f094d

Test: add timeout support in test_coreclr.csproj

c3cbffe

Test: the meta runner now use the same random seed for all tests

4253a30

yatli suggested changes May 8, 2017

View reviewed changes

leasunhy added 7 commits May 10, 2017 20:11

Test: add naive result aggregator

336aaad

Test: add naive result xml synthesizer for tests that exited abnormally

808cea5

Test: fix bugs in meta runner

879a28b

Test: update test_coreclr.csproj

2725cf5

Test: fix bug in the meta runner

e01de4b

Test: fix start-time, end-time and duration for tests that exited abn…

f5d8acb

…ormally

Test: meta runner - fix the issue of file name and add messages for t…

133bf14

…imeouts

yatli approved these changes May 11, 2017

View reviewed changes

yatli merged commit 9b7545a into microsoft:test May 11, 2017

leasunhy deleted the test-nunit branch April 18, 2018 15:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate from xUnit to NUnit #79

Migrate from xUnit to NUnit #79

leasunhy commented Apr 28, 2017 •

edited by yatli

Loading

msftclas commented Apr 28, 2017

yatli commented Apr 28, 2017

yatli Apr 28, 2017

leasunhy Apr 30, 2017

leasunhy commented Apr 30, 2017 •

edited

Loading

yatli commented May 1, 2017

yatli May 2, 2017

leasunhy May 2, 2017

yatli May 2, 2017

leasunhy May 2, 2017 •

edited

Loading

yatli May 2, 2017 •

edited

Loading

leasunhy May 2, 2017

leasunhy commented May 4, 2017 •

edited

Loading

leasunhy commented May 4, 2017

yatli commented May 4, 2017 •

edited

Loading

yatli May 8, 2017

leasunhy May 8, 2017

yatli May 8, 2017

Migrate from xUnit to NUnit #79

Migrate from xUnit to NUnit #79

Conversation

leasunhy commented Apr 28, 2017 • edited by yatli Loading

msftclas commented Apr 28, 2017

yatli commented Apr 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leasunhy commented Apr 30, 2017 • edited Loading

yatli commented May 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leasunhy May 2, 2017 • edited Loading

Choose a reason for hiding this comment

yatli May 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leasunhy commented May 4, 2017 • edited Loading

leasunhy commented May 4, 2017

yatli commented May 4, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leasunhy commented Apr 28, 2017 •

edited by yatli

Loading

leasunhy commented Apr 30, 2017 •

edited

Loading

leasunhy May 2, 2017 •

edited

Loading

yatli May 2, 2017 •

edited

Loading

leasunhy commented May 4, 2017 •

edited

Loading

yatli commented May 4, 2017 •

edited

Loading