fix NeoSystem shutdown order

[ixje]

fix neo-project/neo-modules#786

What I theorize is happening is that while the plugins are disposed (and they remove their Committing/Committed handlers) in the background the main persist loop continues to add blocks and transactions. The plugins (like app log) no longer process these blocks hence we lose data. By first shutting down the "main loop" we ensure plugins don't miss anything.

[superboyiii]

Due to my test, some issues still exist.

1. Throw exception while exit

neo> exit
Unhandled exception. System.InvalidOperationException: Cannot create child while terminating or terminated
   at Akka.Actor.ActorCell.MakeChild(Props props, String name, Boolean async, Boolean systemService)
   at Akka.Actor.ActorCell.AttachChild(Props props, Boolean isSystemService, String name)
   at Akka.Actor.Internal.ActorSystemImpl.SystemActorOf(Props props, String name)
   at Akka.Actor.Inbox.Create(ActorSystem system)
   at Neo.NeoSystem.EnsureStopped(IActorRef actor) in E:\PR-test\neo-PR#2842\neo\src\Neo\NeoSystem.cs:line 206
   at Neo.Plugins.StateService.StatePlugin.Dispose()
   at Neo.NeoSystem.Dispose() in E:\PR-test\neo-PR#2842\neo\src\Neo\NeoSystem.cs:line 171
   at Neo.CLI.MainService.Stop() in E:\PR-test\neo-PR#2842\neo-node\neo-cli\CLI\MainService.cs:line 450
   at Neo.CLI.MainService.OnStop() in E:\PR-test\neo-PR#2842\neo-node\neo-cli\CLI\MainService.cs:line 358
   at Neo.ConsoleService.ConsoleServiceBase.Run(String[] args) in E:\PR-test\neo-PR#2842\neo-node\Neo.ConsoleService\ConsoleServiceBase.cs:line 555
   at Neo.Program.Main(String[] args) in E:\PR-test\neo-PR#2842\neo-node\neo-cli\Program.cs:line 20

2. Some plugins which need persist data like StateService, still losing data while exit or ctrl + C
   I've chosen some random height to verify state root hash after several exit, they showed the wrong hash which means data missing.
   For example:
   My test node

neo> state root 20000
{"version":0,"index":20000,"roothash":"0xe1989c24e4c87c77ed9f6c9a0c9a87cd4f7fac46fc8ca4108cfa409c630ebe20","witnesses":[]}

mainnet seed node

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
        "version": 0,
        "index": 20000,
        "roothash": "0xa776adf30617805e86feccb360e2521cd5eb68cb0a12bc1cfce56109f3940faf",
        "witnesses": []
    }
}

I will take a look at StatesDumper plugin while I finished resyncing full data of mainnet. In previous version, it will surely lose data after being interupted by ctrl + c or exit. See issues here: neo-project/neo-modules#736

[superboyiii]

As what I predicted, StatesDumper still has this issue:


Each exit will cause data lose in StatesDumper.

[ixje]

I've not encountered this shutdown error myself, but somebody else inside coz has so we'll need to investigate further.

[ixje]

It seems like plugins cannot call NeoSystem.EnsureStopped after ActorSystem.Dispose()

 public void Dispose()
        {
            EnsureStopped(LocalNode);
            // Dispose will call ActorSystem.Terminate()
            ActorSystem.Dispose();
            ActorSystem.WhenTerminated.Wait();
            foreach (var p in Plugin.Plugins)   <- TooLate
                p.Dispose();
            HeaderCache.Dispose();
            store.Dispose();
        }

Moving it up like shown next resolves that exception.

 public void Dispose()
        {
            EnsureStopped(LocalNode);
            // Dispose will call ActorSystem.Terminate()
            foreach (var p in Plugin.Plugins)   <- OK
                p.Dispose();
            ActorSystem.Dispose();
            ActorSystem.WhenTerminated.Wait();
            HeaderCache.Dispose();
            store.Dispose();
        }

but then blocks still get persisted e.g.

Persisting block 0
App log committing 0
App log committed 0
NEO-CLI v3.5.0  -  NEO v3.5.0  -  NEO-VM v3.5.0

neo> Persisting block 1
App log committing 1
App log committed 1
Persisting block 2
App log committing 2
App log committed 2
Persisting block 3
App log committing 3
App log committed 3
exit
Persisting block 4  <- OH NO (this is printed in Blockchain.Persist())

Process finished with exit code 0.

I can fix this by adding an EnsureStopped(blockchain) like below

 public void Dispose()
        {
            EnsureStopped(LocalNode);
            EnsureStopped(Blockchain);
            // Dispose will call ActorSystem.Terminate()
            ActorSystem.Dispose();
            ActorSystem.WhenTerminated.Wait();
            foreach (var p in Plugin.Plugins)
                p.Dispose();
            HeaderCache.Dispose();
            store.Dispose();
        }

to get

Persisting block 0
App log commiting 0
App log committed 0
NEO-CLI v3.5.0  -  NEO v3.5.0  -  NEO-VM v3.5.0

neo> Persisting block 1
App log commiting 1
App log committed 1
Persisting block 2
App log commiting 2
App log committed 2
exit
Persisting block 3
App log committing 3
App log committed 3
Persisting block 4
App log committing 4
App log committed 4

Process finished with exit code 0.

Now I still have to fix the exception that this causes when syncing from an offline chain.

[ixje]

@superboyiii this should be it. The exception during offline chain sync I was seeing was due some change I did to the offline syncing system and does not apply to the default system. Can you help test this please? :)

[superboyiii]

@superboyiii this should be it. The exception during offline chain sync I was seeing was due some change I did to the offline syncing system and does not apply to the default system. Can you help test this please? :)

Sure. I'm doing that.

[superboyiii]

@ixje Nice fix! The exception and the inconsistent StateService issue has gone. For StatesDumper, I think it's another issue.

[superboyiii]

For StatesDumper, I think the issue comes from neo-project/neo-modules#568 (comment)

[vncoelho]

Good fix @ixje

[dusmart]

Thanks. This fixed my problem, too. #2624