db_write hook chaining

[ZmnSCPxj]

db_write would appreciate chaining: more backups == better.

More specifically, db_write would appreciate parallel chaining. Each db_write plugin really just returns an "ACK"/"NAK"; if any "NAK", we abort. Since a good backup plugin would first check that the replica has confirmed complete receipt of a set of queries, it would block until it does so. If we did chaining the "normal" way, then having two backup plugins would have us hand over the data to one of them, wait for its replica to confirm receipt, then hand over the data to the other, and wait for its replica to confirm receipt.

So, we should send out db_write to all hooked plugins in parallel.

We could modify db_write, which has its own io_loop over in plugin_exclusive_loop, to instead have an exclusive loop over multiple plugins. We iterate over all registered plugins and create requests to them (it would be nice if we could make the JSON request once and just clone it; is there a function already for that?), set a counter to the number of registered plugins, then enter an exclusive loop for all of them. The db_hook_response, if the response is {"result":"continue"} then decrements that counter and only breaks out of the loop if the counter is 0, thus resuming operations.

Of course, if one of the replicas goes down, the others will have replicated a set of queries that the downed replica does not. This should be fine. If the downed replica is brought up and then lightningd is restarted then the version will be repeated and the other replicas will reverse their last transaction; if the lightningd is recovered from any of the surviving replicas then it gets to see that transaction but will behave as if it had an abnormal shutdown after the transaction but before it could send any messages to peers, which we should also handle properly (the protocol provides for this).

As best I can determine (so far) it seems safe to binary-copy an SQLITE3 database during a time when db_write is blocked waiting for a response from a plugin.

A parallel invocation of db_write does mean that the newfangled before and after specs are pointless for the db_write hook.