Akka.Persistence.TestKit implementation

[Aaronontheweb]

Ported from petabridge/Akka.Persistence.Extras#58

Akka.Persistence.TestKit Requirements

This is an outline for an "Akka.Persistence.TestKit" NuGet package that can be used by Akka.NET developers to help test for possible errors that may come up in the course of using Akka.Persistence, such as:

1. Rejected messages due to serialization errors;
2. Write failures due to the journal or snapshot store being offline; and
3. Recovery errors as a result of serialization or connectivity issues.

These fake journal and snapshot store implementations should be programmable, so if we take a typical Arrange-Act-Assert pattern and factor our "FailingJournal" or "FailingSnapshotStore" into it:

public class MyTestKitClass : Akka.TestKit.Xunit2.Testkit{
	private readonly FailingJournal _failingJournal

	public MyTestKitClass() 
		: base(FailingJournal.DefaultConfig()) // need FailingJournal HOCON 
	{ 
		// need to grab an instance of the FailingJournal for programming purposes
		_failingJournal = PersistenceExt.Get(Sys);
	}

	public async Task MyFakeUnitTest(){
		// arrange
		var myPersistentActor = Sys.ActorOf(Props.Create(() => new MyPersistActor()), "p");
		myPersistentActor.Tell("write"); // write one message to journal while it's still working
		ExpectMsg("ack"); // message gets processed

		// terminate actor
		await myPersistentActor.GracefulStop();		

		// act

		await _failingJournal.BeginFailingWrites(); // simulate journal network disconnect		

		// assert
			
		// re-create a persistent actor - should fail
		EventFilter.Exception<RecoveryException>().ExpectOne(() => {
			var myPersistentActor2 = Sys.ActorOf(Props.Create(() => new MyPersistActor()), "p");
			
			// actor should stop as a result of recovery failure
			Watch(myPersistentActor);
			ExpectTerminated(myPersistentActor);
		});
	}
}

That's the general idea - make it easy to progam the behavior the journal and shapshot store in Akka.Persistence prior to actually using it.

Lexicon and Terminology

To make it easier to understand the requirements going forward, here are some technical terms we're going to use throuhout this document:

• FailingJournal - this is the journal that can be programmed to fail altogether or reject persistent messages.
• FailingSnapshotStore - same idea as the FailingJournal, but for the SnapshotStore.
• "Rejected message" - this means that the connection to the underlying journal is intact, but a specific message was rejected because it couldn't be serialized.
• "Failed message" - this means the message couldn't be written to the journal because the underlying datastore wasn't available due to a network partition.
• "Rejected snapshot" - same idea as "rejected message," but for the snapshot store.
• "Failed snapshot" - same idea as the "failed message," but for the snapshot store. Should trigger a SnapshotWriteFailure message.

Functional Requirements

The Akka.Persistence.TestKit should be built against the current dev branch and released as part of the v1.4.0 release effort.

FailingJournal and FailingSnapshotStore APIs

The FailingJournal and FailingSnapshotStore classes APIs must enable the end-user to do the following:

1. Toggle the journal/snapshot store's entire availability on and off - writes and recovers will both be rejected. If this method returns a Task, by the time that Task completes the toggle operation must have completed.
2. Toggle only the read or the write side of the journal/snapshot store's availability on and off - writes can be rejected but reads will succeed. If this method returns a Task, by the time that Task completes the toggle operation must have completed.
3. Set a filter on the journal/snapshot store to reject or fail messages (this should be a method parameter) on write or reead (should be a parameter) only if they match a certain predicate function specified by the end-user. That predicate will remain in-place on the journal or snapshot store until reset by the user. If this method returns a Task, by the time that Task completes the predicate must be securely in-place.
4. Unset all filters on the journal / snapshot store, if any were previously set by the end-user. If this method returns a Task, by the time that Task completes the toggle operation must have completed.

Enabling FailingJournal and FailingSnapshotStore inside an Akka.NET TestKit Test

The FailingJournal and the FailingSnapshotStore should wrap around the underlying MemoryJournal and MemorySnapshotStore and use them for underlying storage.

But in order to activate either one of these components inside a TestKit test, the end-user must pass in the embedded HOCON built into the FailingJournal.DefaultConfig() or FailingSnapshotStore.DefaultConfig() methods into the TestKit base class constructor, like this:

public class MyTestKitClass : Akka.TestKit.Xunit2.Testkit{
	private readonly FailingJournal _failingJournal

	public MyTestKitClass() 
		: base(FailingJournal.DefaultConfig()) // need FailingJournal HOCON 
	{ 
		// need to grab an instance of the FailingJournal for programming purposes
		_failingJournal = PersistenceExt.Get(Sys);
	}

	// tests
}

Testing Requirements

Need to cover all functional requirements with unit tests, in order to make sure that the API requirements are all met by the implementation. Expand as needed along with the API surface area.

[Aaronontheweb]

To be clear - this is different than the Akka.Persistence.TCK which is used for testing Akka.Persistence plugins. This is for _testing actors built with Akka.Persistence.

[Aaronontheweb]

A better way of designing this API might be to use scoping operators similar to what we do with the TestKit.EventFilter:

 public class MyTestKitClass : Akka.TestKit.Xunit2.Testkit{
	private readonly FailingJournal _failingJournal

	public MyTestKitClass() 
		: base(FailingJournal.DefaultConfig()) // need FailingJournal HOCON 
	{ 
		// need to grab an instance of the FailingJournal for programming purposes
		_failingJournal = PersistenceExt.Get(Sys);
	}

 public void TestForNetworkFailure(){
	
	var peristActor = Sys.Actorof( Props.Create(()=> new aPersitActor()), "Persitor");
	Watch(persitActor);

	persistActor.Tell("Message");
	ExpectMsg("Ack");
	
	FailingJournal.NetworkFailure(FailureOn.Write,()=>
	{
		persistActor.Tell("Message");
		ExpectTerminated(persistActor); // Actor should be terminated due to persistence failure
	});
	
	var newPeristActor = Sys.Actorof( Props.Create(()=> new aPersitActor()), "Persitor");
	newPersistActor.Tell("Message"); 

	ExpectMsg("Ack"); // newPersistActor should be able to persist messages to the journal again
	

		
 }

Before the lamba inside the FailingJournal.NetworkFailure(FailureOn.Write,()=> { }); starts, the journal is set to fail writes altogether, which should cause persistent actors to crash. But those actors could still recover. Once that lambda exits, the journal is restored back to normal working behavior again.

[Aaronontheweb]

closed via #3881 and #3889