Tracking issue: libp2p introspection support (proof-of-concept)

[raulk]

This umbrella issue enumerates all the pull requests and linked issues that make up the initial introspection proof-of-concept for go-libp2p.

The project is affectionately codenamed Phantom Drift 👻🌊, and we'll be open sourcing the UI and widget library soon. A sneak peek:

Scope

The initial proof-of-concept comprises these components and concepts on the libp2p backend side of the fence:

• an Protobuf introspection schema that libp2p implementations adhere to to offer introspection ability.
• a WebSockets-based introspection protocol that uses the schema and offers a set of RPCs and streaming operations to fetch the current state of the stack and the events circulating through the eventbus.
• an introspectable Host: a Host that renders itself to introspection by offering an introspection endpoint that speak the above protocol.
• an introspector component that does the heavy-lifting of sourcing data from the inner libp2p modules (e.g. swarm, DHT, etc.) to expose over the introspection endpoint.
• REPL support for quick testing.

On the frontend/application/UI side of things:

• the libp2p observer shell: a web application onto which you can load libp2p observation widgets, load file dumps of introspection data, or connect to a live libp2p instance to visualise real-time, or travel back and forth in time to view the state of the host at that point. https://github.com/libp2p/observation-deck
• the widget toolkit: a JS library to develop observation widgets against the libp2p introspection protocol. https://github.com/libp2p/observer-toolkit
• various widgets to playing with out of the box.

go-libp2p work tracker

• libp2p/go-libp2p-core:
  • (Long-lived branch): Introspection capability for go-libp2p go-libp2p-core#112
  • Introspection support: add schema, interfaces, data provider system go-libp2p-core#142
  • eventbus: add wildcard subscription type; getter to enumerate known types go-libp2p-core#153
  • events: add a generic DHT event. go-libp2p-core#154
  • experimental introspection support go-libp2p-core#159
• libp2p/go-libp2p-introspector: may merge into go-libp2p.
  • [WIP] Introspection 2.0 changes for Client signalling, events and DHT introspection go-libp2p-introspector#11
  • potentially move into go-libp2p.
• libp2p/go-libp2p-swarm:
  • (Long-lived branch): Introspection – harvest data from go-libp2p-swarm go-libp2p-swarm#159
  • introspection support: source connection and streams data go-libp2p-swarm#204
  • ID() method on connections and streams + record opening time go-libp2p-swarm#224
• libp2p/go-libp2p:
  • (Long-lived branch): Introspection capability for go-libp2p #792
  • introspection support: implement host.IntrospectableHost and wire up the introspector #885
  • experimental introspection support -- Project Phantom Drift PoC #964
• libp2p/repl:
  • new iteration of introspection logic repl#5
• libp2p/go-eventbus:
  • Support wildcard subscriptions + getter for known types go-eventbus#34
  • Support wildcard subscriptions + getter for known types (optimized) go-eventbus#40
• libp2p/go-libp2p-kad-dht:
  • [Long Lived Brach and heavy WIP] DHT Introspection work. Do NOT judge the programmer yet. go-libp2p-kad-dht#590

For reference, here's the initial issue where this work was initially introduced: #775

[raulk]

After reviewing the solution end-to-end, I'm proposing that we make a few architectural changes.

• I don't like the fact that modules (e.g. swarm) are responsible for populating the introspection schema. I think they should just return go-libp2p-core objects, and the introspector should translate to the introspection schema.

• That model also creates symmetry -- the data you can obtain from introspection is the same data that's in the libp2p core model. Otherwise, we can get into a paradoxical situation where an outside observer (introspection UI) would be capable of obtaining more information about the system than an internal observer (via the core API).

• I'm going to rejigger the data provider system / introspector accordingly.

• I might also move the introspector into go-libp2p -- to align with our pursuit of codebase simplification. The introspector is one of those "replaceable, not modular" components. 99% of users will use the default introspector; the other 1% can implement the interface and inject the custom implementation via the constructor option.

[marten-seemann]

Closing this. See #775 (comment) for motivation.