Libp2p introspection

[aarshkshah1992]

This is a long-lived meta-issue to track the Libp2p introspection work. Please note that this work is heavy WIP & the details below will be subject to frequent change as we progress with this feature.

Goal & Scope

• Give users the ability to capture the state of a libp2p network from the point of view of a single node by implementing an "introspection protocol" / data format.
• The protocol provides a point in time snapshot of a libp2p node allowing users to access stats including but not limited to peers connected, data being sent throughout the network, DHT meta information such as the number of hops required to complete a query etc etc.
• The protocol should eventually evolve to provide information about all libp2p subsystems such as Swarm, DHT, Pubsub, Peerstore etc. allowing users to build interactive visualizations or diagnostics tooling that provides a holistic view of the state of a node & also how it changes over time.

Design

• In the libp2p spirit of modularity, we want to make the implementation modular so that users can plugin their own implementation of how to:
  * Build the snapshot of the node/introspection protocol by allowing libp2p subsystems to register themselves as state providers & then sourcing the data from these providers.
  * Serve the data sourced above to clients over the wire. For eg: A websocket server that serves the data as protobuf.

To this end, we have currently defined two interfaces:

// Endpoint is the interface to be implemented by introspection
// endpoints/servers.
//
// An introspection endpoint exposes introspection data over the wire via a
// protocol and data format, e.g. WebSockets with Protobuf.
type Endpoint interface {
 // Start starts the introspection endpoint. It must only be called once, and
 // once the server is started, subsequent calls made without first calling
 // Close will error.
 Start() error

 // Close stops the introspection endpoint. Calls to Close on an already
 // closed endpoint, or an unstarted endpoint, must noop.
 Close() error

 // ListenAddrs returns the listen addresses of this endpoint.
 ListenAddrs() []string
}

AND

Introspector allows other sub-systems/modules to register
// metrics/data providers AND also enables clients to fetch the current state of
// the system.
//
// Introspector implementations are usually injected in introspection endpoints
// (e.g. the default WebSocket endpoint) to serve the data to clients, but they
// can also be used separately for embedding or testing.
type Introspector interface {
	// RegisterDataProviders allows sub-systems/modules to
	// register callbacks that supply introspection data.
	RegisterDataProviders(p *DataProviders) error

	//  FetchFullState returns the full state of the system, by
	// calling all known data providers and returning a merged cross-cut of the
	// running system.
	FetchFullState() (*introspection_pb.State, error)
}

Implementations of both these components can then be injected into an "Introspectable" host which can expose them to it's subsystems(for sourcing the data) & clients(for serving the data). This detail is captured by an additional interface that a host which is interested in being introspected must implement:

// IntrospectableHost is implemented by Host implementations that are
// introspectable, that is, that may have introspection capability.
type IntrospectableHost interface {
	// Introspector the introspector, or nil if one hasn't been registered. With
	// it, the call can register data providers, and can fetch introspection
	// data.
	Introspector() introspection.Introspector

	// IntrospectionEndpoint returns the introspection endpoint, or nil if one
	// hasn't been registered.
	IntrospectionEndpoint() introspection.Endpoint
}

Current State

• Since the work is heavy WIP, we have long lived branches open across libp2p for this work
• Interfaces, common abstractions & the current introspection protocol have been defined in go-libp2p-core in (Long-lived branch): Introspection capability for go-libp2p go-libp2p-core#112.
• Default working implementations for the Endpoint & Introspector interfaces have been implemented in the newly created go-libp2p-introspector repo. This currently enables clients to fetch the available introspection data in protobuf over a Websocket server.
• The libp2p BasicHost implements the IntrospectableHost interface and can be introspected. PR is at (Long-lived branch): Introspection capability for go-libp2p #792
• We currently ONLY provide go-libp2p-swarm metrics as implemented in (Long-lived branch): Introspection – harvest data from go-libp2p-swarm go-libp2p-swarm#159. Some of the swarm metrics required by the current introspection protocol are still not available and the same have been documented as TODOs in the PR.
• We are building a UI("visualizer UI ") that will provide interactive visualisations for the metrics that are currently available. The setup for the same for will be documented here once it's ready.

See it in action

• Hop over to https://github.com/libp2p/repl & select the "Introspect Host" Option.

Next Steps

• Keep refining the existing interfaces, implementations & the Introspection protocol as we get a better understanding of how users want to use this feature.
• Keep refining the visualizer UI to enable better user experience & diagnostics
• Finish sourcing all the Swarm metrics
• Source metrics from the DHT as defined in the Introspection protocol
• Define the protocol & implement the sourcing for other libp2p subsystems such as Pubsub, Peerstore etc.

[momack2]

Hey @aarshkshah1992 - can you add a fleshed out description to this with goals, scope, todos, etc?

[aarshkshah1992]

@raulk Please can you take a look at fleshed out issue & let me know what you think ?

[marten-seemann]

We've decided to go a different way by adding Prometheus metrics for all of the libp2p subsystems. I don't see us pick up this work any time soon, in fact, we've removed the unused introspection package in #1978.

Our current metrics effort is tracked in #1356, and we've just added Prometheus metrics for the swarm in #1973. The resource manager has already been exporting metrics for a while.
Admittedly, metrics don't capture the full use case of the introspector, for example, it's harder to react to changes programmatically. Arguably, users should use libp2p events and not an introspection.