Canonicalizing Dual Execution for Node/Application-specific Processing

[vyzo]

Dual Execution as Observable Computation

With the PoC in #8841, we already support dual execution whereby canonical actors are redirected to a recplacement actor and executed in parallel for side-effect.
This was outlined as a debugging enhancement in filecoin-project/ref-fvm#592, which allows us to obtain debug traces for execution and debug extant actors and test new versions.

However, It has become apparent that there is a more general usage patter for dual code execution: business logic that monitors actor execution. Prior to the FVM, this was typically accomplished by operators running custom specs-actors code. This is no longer possible due to the strict code CID mapping, but the need is still there: there are legitimate application use cases whereby business logic is executed in parallel with the canonical actor code, in order to observe it, trigger events, and general node/application-specific logic.

We would like to support this mode of operation, and dual execution is the ideal mechanism for this. The node operator supplies a dual bundle for builtin actors, which runs and performs necessary business logic. The one thing that is missing, is output.

Considerations

Currently, the output is only emitted through the logging debug system call, and goes to the stderr. In order to facilitate processing of dual execution, its output must be captured and subsequent processed in some manner.

There are three ways to prduce such output:

• Save the dual actor state, return the state tree.
• Save the dual actor execution return value.
• Introduce a record system call that records output emitted by dual execution.

In terms of mechanics, and in order to avoid bloating the blockstore, it appears that options (2) and (3) are the best ways to proceed. Dual actor message return is probably the easiest way forward for a PoC implementation.

Experimental PoC

In order to experiment with these mechanisms and provide the grounds for further discussion whereby we can arrive at a robust solution for the problem, we propose to implement a PoC that implements (2) and/or (3) and provides a simple mechanism to capture it for the application to process.

Possible processing avenues:

• save to disk in json log form.
• use an external process to pass output to through a pipe/socket (in json log form probably).
• Store in memory/disk and provide a JSON-RPC access api.

[Stebalien]

FYI, @mriise is already working on a method for saving assets that could be used by option 3: filecoin-project/ref-fvm#616.

However, I'm not sure if that's the right fit here. It sounds like we want structured logging, right?

[mriise]

if its just a dump at the end of the process, store_artifact can just be passed whatever, though it just stores raw bytes to disk (in its own subdirectory per-invocation), so any sort of serialization of the logs to json or whatever would need to be done inside the actor.

[vyzo]

yeah, i also think we want structured logging, but maybe unstructured will also work.

[arajasek]

Thanks for the idea, @vyzo, and for getting the discussion started!

From a matter of design, I think 3 is what we want -- the ability to log and query custom events that are of special interest to the user. Structured logging would be the nicest answer here, and shouldn't be too much work to prototype (and a prototype is honestly good enough here).

From a matter of priority, I'm a little less clear. This wouldn't really be solving a regression introduced by the FVM -- if there is one at all, I think #8841 solves that. This is introducing new functionality that we have some signal that users would like. I'd be very happy to see this land in Lotus, and would get some use of it myself, but it might not be the highest priority based on the M2 goals.

Having said that, if you feel motivated to build out what this might look like (at least within the FVM itself), I'd love to see it!