NUnit tests fail if the fixture contains more than one test

[garethbudden]

I've hit a problem when trying to use the NUnit test suite, given the following example...

public class NUnitTestFixture
{
    [Test]
    public void SayHello()
    {
        ActorOf<Parrot>("test").Tell(new SayHello());

        ExpectMsg<Hello>();
    }

    [Test]
    public void SayGoodbye
    {
        ActorOf<Parrot>("test").Tell(new SayGoodbye());

        ExpectMsg<Goodbye>();
    }
}

When I run the tests the first succeeds but the second fails with Akka.Actor.InvalidActorNameException : Actor name "test" is not unique!

If I instead run the same tests but with XUnit then they succeed, I believe this could be related to #972, which says that NUnit keeps the ActorSystem around for the full test fixture, but XUnit creates a new ActorSystem for each test.

[rogeralsing]

This is definitely a bug in the NUnit TestKit, this package is pretty new and thus not very well tested in the field.
We use XUnit TestKit internally.

[garethbudden]

One option would be to wipe the ActorSystem after each test run, as was mentioned in #972 this may not be performant, on the other hand it would be more consistent with how XUnit works. Or is there a better way to handle this entirely?

[rogeralsing]

Yes, imo, we should spin up a new instance of the system and init all the defaults after each test.
This is after all what happens in the XUnit Testkit, each parallel test has its own system and infrastructure

[Aaronontheweb]

Is this really a bug? The current design seems right to me - the actor system should be shared across all of the tests in a fixture. Our multi-node tests would all break if this wasn't the case.

[Aaronontheweb]

Ah, I take that back about the multi-node tests - that's not the case there.

[rogeralsing]

If the system was shared between tests, we would end up with a gazillion racy tests with completely undeterministic behavior

[Aaronontheweb]

Where inside the XUnit testkit does the actor system get shut down after each test currently?

(edit: I tried looking but it isn't obvious to me)

[rogeralsing]

Yes, there is one instance per test in XUnit, all isolated

[Aaronontheweb]

Yes, there is one instance per test in XUnit, all isolated

Can you walk me through that inside the code of the XUnit fixture itself? Like what's the call chain that actually makes this happen per-test? Everything looks per-fixture to me.

[rogeralsing]

        [Fact]
        public void Unique1()
        {
            var hash= Sys.GetHashCode();
            Console.WriteLine(hash);
        }

        [Fact]
        public void Unique2()
        {
            var hash = Sys.GetHashCode();
            Console.WriteLine(hash);
        }

Run that in a testkit test and you will see that the systems in the two tests get different hash codes.
XUnit does weird scheduling so you cant actually step from the test init to the test methods

Or, just set a BP in the test's ctor and you will see that its called as many times as there are tests running

[Aaronontheweb]

Ok, got it. In that case I guess we should add a cleanup attribute to a method inside the NUnit testkit?

[garethbudden]

I thought I may take a stab at this one.

I've had a quick look and I think the simplest thing to do may be to take all the private fields from TestKitBase.cs and put them into a new private class called TestKitBaseState.

The XUnit implementation would be able to create and dispose this class using the constructor and dispose method, as it does now. The NUnit implementation would be able to create a new TestKitBaseState instance per unit test using the SetUp attribute, and dispose of the state class using the TearDown attribute.

I've not used MsTest myself but a quick google scan implies the same issue as NUnit, and could be fixed with the same approach.

Before I make an attempt at implementing it I thought I'd check if this sound reasonable to you?