Implement a streaming trace capability

[benjamincburns]

Numerous users have complained about the size of transaction traces (from debug_traceTransaction), and in fact forks of ganache-core have been created to deal with this issue in various ways.

From my perspective, the problems with debug_traceTransaction today are:

1. entire response must be buffered before it can be sent
2. getting a trace requires changing state (you're tracing a transaction after it has been mined)
3. limited ability for the user to choose which data they want to receive

To fix this we should implement a feature which allows for users to:

1. specify exactly what data they wish to receive in their trace
2. receive trace data as each step is processed
3. trace a previously ran transaction
4. trace arbitrary EVM bytecode without changing state

One proposal that seems to meet most of these criteria is #47, however #47 would have us calling eval on unsanitized code sent to us via a network interface. This would severely limit the contexts in which it would be safe to deploy ganache.

A counter-proposal might be to build a tracing mechanism which leverages the existing eth_subscribe functionality. We could allow for users to specify at a fine level of detail what trace information they'd like to receive, whether they'd like to receive traces on calls to eth_call, whether they'd like to receive traces for a specific transaction hash, transactions to or from specific addresses, or all transactions executed by the node until eth_unsubscribe is called.

We might want to use GraphQL for this. It supports subscriptions, and allows for robust queries against self-describing schema. And then there's also the @defer feature.

All that said, this issue isn't intended to be prescriptive. The solution we should implement is the simplest one that achieves the goals stated above in a secure, efficient manner that's easy for our users to understand and use.

[benjamincburns]

Just so it doesn't get lost, here's some more talk on GraphQL features that could be useful to us here.

[LogvinovLeon]

I think - using the eval is not the only way. Geth somehow executes JS code without having security vulnerabilities (and they store user's private keys, so the stakes are higher there). They just execute it in a sandbox and it's able to access just a very limited number of variables. I think we should try to keep the clients API consistent.

[davidmurdoch]

@LogvinovLeon, I agree with you. I don't think we've exhausted all possible solutions for running user code safely just yet; there are some node-js based VMs out there, or even something like running it in chromium via https://github.com/GoogleChrome/puppeteer (though this is a HUGE dependency, so I seriously doubt this could be a solution itself).

I know node comes with a vm module and there is at least 1 library that uses it called vm2 that claims will make it "Secure!".

Basically, we can create a node VM that has a global object with no properties except for the ones you give it. There is no require here, and its execution can be time-limited -- so while(true){} doesn't lock thing up. I think you can put memory limits on it as well, mitigating one potential DoS attack vector, if the RPC endpoint was opened up externally.

Go can do it safely because the JS interpreter used there (otto) doesn't have the ability to access the filesystem or IO, as far as I know.

This is definitely something I've been thinking about a lot and look forward to implementing in the near future.