concept of a miner DMZ

[raulk]

link: https://en.wikipedia.org/wiki/DMZ_(computing)

Problem

The lotus miner is a specialised process with a several purposes that seem to be at odds with each other when it comes to network/external exposure/isolation requirements:

1. To produce new blocks in the blockchain (by running leader election, winning PoSt, and broadcasting the block).
2. Managing storage by onboarding new sectors, and proving them when challenged (Window PoSt).
3. Accepting and sealing deals.

For 1 and 2, the miner doesn't need to be exposed openly on the p2p network. These processes can be carried out in relative isolation, proxying via a companion daemon, which is what happens now. With this architecture, the miner process enjoys certain protection from the p2p network.

But for 3, the protocol currently expects the miner to be directly reachable/diallable. This is at odds with 1 and 2 above.

Preliminary network diagnostics show that many miners choose to not participate in the DHT nor advertise multiaddrs in their miner actor, therefore they opt to be not diallable and, thus, accept no deals.

Discussion

I think we need an official concept of a miner DMZ, which is a specialised node that takes the role of the companion daemon, but also enhances it with a deals bridge/depot functionality.

This way, the DMZ would take the responsibility of being exposed in the network, accepting deals on behalf of the miner with a predefined policy, participating in the storage market, etc. while the core miner would stay isolated from the network.

The miner DMZ would also own keeping the storage market actor updated with the deal status.

This would also protect the miner from bugs/inefficiencies in the deals, markets, data transfer components, like #4877. For scaling deals, a miner could run many replicated miner DMZ nodes fronting a single miner.

[hannahhoward]

@raulk for historical reference, in the design of Filecoin, @jbenet specifically called out the need for seperate network properties among different components of the system. So, for example, markets/data transfer wants high bandwidth connections with less need for security, while other components want the opposite. This is part of why markets & data transfer live in it's own repo across it's own API boundary -- the thought is these can be seperated into their own process boundary and run on their own libp2p host. It's never actually come together, but this was a motivator for the strong seperation of this component. Actually, I believe there is a project still ongoing the Rust folks are working on that put markets in their own process. -- https://github.com/ChainSafe/go-fil-markets-interface

[hannahhoward]

^^ upon further reflection I think the repo referenced above is just for a seperated markets client.

[raulk]

@hannahhoward This is very much reminiscent of an actor-oriented programming model like Erlang's or Akka. I am personally a strong proponent of such an architecture for blockchain systems that are composed of various moving parts / processes, many of which need to work together, but not in perfect synchrony or lockstep. Where faults of different kinds are expected and tolerated at different levels, and each component is designed with clean inputs and outputs, under a defensive, reactive, async, backpressured programming model, and with resilience as a top requirement.

Unfortunately, breaking off components in separate repos is a start, but if they end up integrated in a single process, that setup creates more complexity than value :-(

I do understand that that this code layout was intended as a stepping stone, but the delta between the original vision and reality is quite large.

I'm not sure I see the value of running each process that comprises a logical identity as a separate physical libp2p identity, but maybe I need to be illuminated.

[hannahhoward]

yea, @raulk there is still an indirection layer in markets which enables a process seperation -- markets have "Node adapters" -- a defined API they rely on for talking to the nodes, that is seperate from the Lotus API (but in the lotus market adapter implementations, obviously calls the lotus API a lot). Anyway, I'm just pointing out there is a clear point at which you can insert a process boundary, and that's been maintained for a reason. whether it's relevant to actually implementing this, I have no idea.

[stuberman]

I hope this gets some traction. As I see the deal volume and traffic increasing, even after reducing the connection manager settings, my ATT router struggles to stay online, causing me to go offline and reboot the router several times each day. When deal proposals were low over the holidays this problem went away. I think if I could proxy P2P and data transfer traffic this would not be so burdensome to the network infrastructure by using a cloud hosted proxy which has far greater capacity.

[dineshshenoy]

This is great - I think there's a similar equivalent separation/DMZ that makes sense on the client side. Calculating the CommP or staging the data can be separated from the markets/deal-making components - this is especially true since the same data is often involved in multiple deals (for redundancy or geographic diversity)

[raulk]

Quick update on this issue to let the community know that this work is now well underway.

• The bulk of the work is defined here in this project definition.
• Talk on the Evolution of a Lotus Node.
• Work in progress PR by @nonsense to address the first step of the refactor: the miner and markets process segregation.