From ecdc6801a455c961db66ad929085fa3bc4426786 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Thu, 10 Nov 2022 17:03:39 -0800 Subject: [PATCH 1/7] First pass at http spec --- http/README.md | 142 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 http/README.md diff --git a/http/README.md b/http/README.md new file mode 100644 index 000000000..04fe31a05 --- /dev/null +++ b/http/README.md @@ -0,0 +1,142 @@ +# libp2p over HTTP + +| Lifecycle Stage | Maturity | Status | Latest Revision | +| --------------- | ------------- | ------ | --------------- | +| 1A | Working Draft | Active | r0, 2022-11-10 | + +Authors: [@marcopolo] + +Interest Group: [@marcopolo], [@mxinden], [@marten-seemann] + +[@marcopolo]: https://github.com/mxinden +[@mxinden]: https://github.com/mxinden +[@marten-seemann]: https://github.com/marten-seemann + +# Table of Contents +- [Context](#context) + - [Why not two separate stacks?](#why-not-two-separate-stacks) + - [Why HTTP rather than a custom request/response protocol?](#why-http-rather-than-a-custom-requestresponse-protocol) +- [Implementation](#implementation) + - [HTTP over libp2p streams](#http-over-libp2p-streams) + - [libp2p over plain HTTPS](#libp2p-over-plain-https) + - [Choosing between libp2p streams vs plain HTTPS](#choosing-between-libp2p-streams-vs-plain-https) +- [Implementation recommendations](#implementation-recommendations) + - [Example – Go](#example--go) +- [Prior art](#prior-art) + +# Context + +HTTP is everywhere. Especially in CDNs, cloud offerings, and caches. + +HTTP on libp2p and libp2p on HTTP are both commonly requested features. This has +come up recently at IPFS Camp and especially in the [data transfer track]. One +aspect of the discussion makes it seem like you can use HTTP _OR_ use libp2p, +but that isn't the case. Before this spec you could use the HTTP protocol on top +of a libp2p stream (with little to no extra cost). And this spec outlines how to use libp2p _on top of_ HTTP. + +This spec defines a new libp2p abstraction for stateless request/response +protocols. This abstraction is notably nothing new, it is simply HTTP. Being +HTTP, This abstraction can run over a plain TCP+TLS HTTP (henceforth referred to +as _plain https_) or on top of a libp2p stream. + +## Why not two separate stacks? + +Having libp2p as the abstraction over _how_ the HTTP request gets sent gives developers a lot of benefits for free, such as: + +1. NAT traversal: You can make an HTTP request to a peer that's behind a NAT. +1. Fewer connections: If you already have a libp2p connection, we can use use that to create a stream for the HTTP request. So that HTTP request will be much faster (You don't have to pay the two round trips to establish the connection) +1. Allows JS clients to make HTTPS requests to _any_ peer via WebTransport. +1. Allows more reuse of the protocol logic, just like how applications can integrate GossipSub, bitswap, graphsync, and Kademlia. +1. You get mutual authentication of peer IDs automatically. + +Get the benefits of libp2p (webtransport, fewer connections, webrtc, nat traversal) + +## Why HTTP rather than a custom request/response protocol? + +HTTP has been around for 30+ years, and it isn't going anywhere. Developers are already very familiar with it. There's is no need to reinvent the wheel here. + +# Implementation + +## HTTP over libp2p streams + +If we have an existing libp2p connection that supports streams, we can run the HTTP protocol as follows: + +Client: +1. Open a new stream to the target peer. +1. Negotiate the `/libp2p-http` protocol. +1. Use this stream for HTTP. (i.e. start sending the request) +1. Close the write side when finished uploading the HTTP request. +1. Close the stream when the response is received. + +Server: +1. Register the `/libp2p-http` protocol. +1. One a new stream speaking `/libp2p-http` pass the stream to an HTTP handler. +1. Close the stream when finished uploading the request. + +## libp2p over plain HTTPS + +This is nothing more than a thin wrapper over standard HTTP. The only thing +libp2p should do here is ensure that we verify the peer's TLS certificate as +defined by the [tls spec](../tls/tls.md). This SHOULD be interoperable with standard HTTP clients who pass a correct TLS cert. For example curl should work fine: + +``` +$ curl --insecure --cert ./client.cert --key ./client.key https://127.0.0.1:9561/echo -d "Hello World" + +Hello World +``` + +## Choosing between libp2p streams vs plain HTTPS + +Implementations SHOULD choose a libp2p stream if an existing libp2p connection +is available. If there is an existing HTTP connection, then implementations +SHOULD use that connection rather than starting a new libp2p connection. If +there is no connection implementations may choose either to create a new HTTP +connection or a libp2p connection or expose this as an option to users. + +# Implementation recommendations + +Each implementation should decide how this works, but the general recommendations are: + +1. Make this look and feel like a normal HTTP client and server. There's no +benefit of doing things differently here, and the familiarity will let people +build things faster. + +1. Aim to make the returned libp2p+HTTP objects interop with the general HTTP ecosystem of the language. + +## Example – Go + +We create a host as normal, but enable HTTP: +``` +h, err := libp2p.New(libp2p.WithHTTP( + HTTPConfig: HTTPConfig{ + EnableHTTP: true, + // Enable + HTTPServerAddr: multiaddr.StringCast("/ip4/127.0.0.1/tcp/9561/tls/http"), + })) +``` + +We can define HTTP Handlers using standard types: +``` +h1.SetHTTPHandler("/echo", func(peer peer.ID, w http.ResponseWriter, r *http.Request) { + w.WriteHeader(200) + io.Copy(w, r.Body) +}) +``` + +We can create a client that accepts standard types +``` + // client is a standard http.Client + client, err := h2.NewHTTPClient(h1.ID()) + require.NoError(t, err) + + resp, err := client.Post("/echo", "application/octet-stream", bytes.NewReader([]byte("Hello World"))) +``` + +# Prior art + +- rust-libp2p's request-response protocol: https://github.com/libp2p/rust-libp2p/tree/master/protocols/request-response +- go-libp2p's [go-libp2p-http] + +[data transfer track]: (https://youtube.com/watch?v=VRn_U8ytvok&feature=share&si=EMSIkaIECMiOmarE6JChQQ) +[rust-libp2p request-response protocol]: (https://github.com/libp2p/rust-libp2p/tree/master/protocols/request-response) +[go-libp2p-http]: (https://github.com/libp2p/go-libp2p-http) \ No newline at end of file From d52be512cb8656fed08b4af3942aba5e655ff512 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Thu, 10 Nov 2022 17:04:42 -0800 Subject: [PATCH 2/7] Add implementation PR link --- http/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/http/README.md b/http/README.md index 04fe31a05..94c8b01b7 100644 --- a/http/README.md +++ b/http/README.md @@ -132,6 +132,8 @@ We can create a client that accepts standard types resp, err := client.Post("/echo", "application/octet-stream", bytes.NewReader([]byte("Hello World"))) ``` +For more details see the implementation [PR](https://github.com/libp2p/go-libp2p/pull/1874). + # Prior art - rust-libp2p's request-response protocol: https://github.com/libp2p/rust-libp2p/tree/master/protocols/request-response From 3b5ee42ab0723dc765eb82f8dfb981aecdfd7379 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 11 Nov 2022 09:04:17 -0800 Subject: [PATCH 3/7] Edits --- http/README.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/http/README.md b/http/README.md index 94c8b01b7..5a7a9e85c 100644 --- a/http/README.md +++ b/http/README.md @@ -29,10 +29,11 @@ Interest Group: [@marcopolo], [@mxinden], [@marten-seemann] HTTP is everywhere. Especially in CDNs, cloud offerings, and caches. HTTP on libp2p and libp2p on HTTP are both commonly requested features. This has -come up recently at IPFS Camp and especially in the [data transfer track]. One -aspect of the discussion makes it seem like you can use HTTP _OR_ use libp2p, -but that isn't the case. Before this spec you could use the HTTP protocol on top -of a libp2p stream (with little to no extra cost). And this spec outlines how to use libp2p _on top of_ HTTP. +come up recently at [IPFS Camp 2022](https://2022.ipfs.camp/) and especially in +the [data transfer track]. One aspect of the discussion makes it seem like you +can use HTTP _OR_ use libp2p, but that isn't the case. Before this spec you +could use the HTTP protocol on top of a libp2p stream (with little to no extra +cost). And this spec outlines how to use libp2p _on top of_ HTTP. This spec defines a new libp2p abstraction for stateless request/response protocols. This abstraction is notably nothing new, it is simply HTTP. Being @@ -44,12 +45,11 @@ as _plain https_) or on top of a libp2p stream. Having libp2p as the abstraction over _how_ the HTTP request gets sent gives developers a lot of benefits for free, such as: 1. NAT traversal: You can make an HTTP request to a peer that's behind a NAT. -1. Fewer connections: If you already have a libp2p connection, we can use use that to create a stream for the HTTP request. So that HTTP request will be much faster (You don't have to pay the two round trips to establish the connection) -1. Allows JS clients to make HTTPS requests to _any_ peer via WebTransport. +1. Fewer connections: If you already have a libp2p connection, we can use use that to create a stream for the HTTP request. The HTTP request will be faster since you don't have to pay the two round trips to establish the connection. +1. Allows JS clients to make HTTPS requests to _any_ peer via WebTransport or WebRTC. 1. Allows more reuse of the protocol logic, just like how applications can integrate GossipSub, bitswap, graphsync, and Kademlia. 1. You get mutual authentication of peer IDs automatically. -Get the benefits of libp2p (webtransport, fewer connections, webrtc, nat traversal) ## Why HTTP rather than a custom request/response protocol? @@ -69,9 +69,10 @@ Client: 1. Close the stream when the response is received. Server: -1. Register the `/libp2p-http` protocol. -1. One a new stream speaking `/libp2p-http` pass the stream to an HTTP handler. -1. Close the stream when finished uploading the request. +1. Register a stream handler for the `/libp2p-http` protocol. +1. On receiving a new stream speaking `/libp2p-http`, parse the HTTP request and pass it to the HTTP handler. +1. Write the response from the HTTP handler to the stream. +1. Close the stream when finished writing the response. ## libp2p over plain HTTPS From cdc729e15769dcade0a1eca010941b7db1038f1b Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 11 Nov 2022 09:08:30 -0800 Subject: [PATCH 4/7] Add to prior art --- http/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/http/README.md b/http/README.md index 5a7a9e85c..960ad9239 100644 --- a/http/README.md +++ b/http/README.md @@ -137,8 +137,9 @@ For more details see the implementation [PR](https://github.com/libp2p/go-libp2p # Prior art -- rust-libp2p's request-response protocol: https://github.com/libp2p/rust-libp2p/tree/master/protocols/request-response -- go-libp2p's [go-libp2p-http] +- rust-libp2p's request-response protocol: https://github.com/libp2p/rust-libp2p/tree/master/protocols/request-response. +- go-libp2p's [go-libp2p-http]. +- The Indexer project uses [go-libp2p-stream](https://github.com/libp2p/go-libp2p-gostream) to do [HTTP over libp2p](https://github.com/filecoin-project/storetheindex/blob/main/dagsync/p2p/protocol/head/head.go). [data transfer track]: (https://youtube.com/watch?v=VRn_U8ytvok&feature=share&si=EMSIkaIECMiOmarE6JChQQ) [rust-libp2p request-response protocol]: (https://github.com/libp2p/rust-libp2p/tree/master/protocols/request-response) From d04725f0eb9b66c570951fe8dba0fec83303a1c1 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 11 Nov 2022 09:16:59 -0800 Subject: [PATCH 5/7] Add historical discussion link --- http/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/http/README.md b/http/README.md index 960ad9239..a8aec52a8 100644 --- a/http/README.md +++ b/http/README.md @@ -137,6 +137,7 @@ For more details see the implementation [PR](https://github.com/libp2p/go-libp2p # Prior art +- Historical discussion: https://pl-strflt.notion.site/libp2p-HTTP-4f7d7ff3350a462a875bbfc41b3d4134 - rust-libp2p's request-response protocol: https://github.com/libp2p/rust-libp2p/tree/master/protocols/request-response. - go-libp2p's [go-libp2p-http]. - The Indexer project uses [go-libp2p-stream](https://github.com/libp2p/go-libp2p-gostream) to do [HTTP over libp2p](https://github.com/filecoin-project/storetheindex/blob/main/dagsync/p2p/protocol/head/head.go). From 1c777be7a7c04a2177305c7af061c6b844424e72 Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 11 Nov 2022 12:42:06 -0800 Subject: [PATCH 6/7] Update http/README.md Co-authored-by: Max Inden --- http/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/http/README.md b/http/README.md index a8aec52a8..c994b491d 100644 --- a/http/README.md +++ b/http/README.md @@ -45,7 +45,7 @@ as _plain https_) or on top of a libp2p stream. Having libp2p as the abstraction over _how_ the HTTP request gets sent gives developers a lot of benefits for free, such as: 1. NAT traversal: You can make an HTTP request to a peer that's behind a NAT. -1. Fewer connections: If you already have a libp2p connection, we can use use that to create a stream for the HTTP request. The HTTP request will be faster since you don't have to pay the two round trips to establish the connection. +1. Fewer connections: If you already have a libp2p connection, we can use that to create a stream for the HTTP request. The HTTP request will be faster since you don't have to pay the two round trips to establish the connection. 1. Allows JS clients to make HTTPS requests to _any_ peer via WebTransport or WebRTC. 1. Allows more reuse of the protocol logic, just like how applications can integrate GossipSub, bitswap, graphsync, and Kademlia. 1. You get mutual authentication of peer IDs automatically. From c7deac56982d5a13d40d9f53018482d2d1c36bfc Mon Sep 17 00:00:00 2001 From: Marco Munizaga Date: Fri, 11 Nov 2022 12:51:39 -0800 Subject: [PATCH 7/7] Add future work section and wrap the text properly --- http/README.md | 50 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 12 deletions(-) diff --git a/http/README.md b/http/README.md index c994b491d..ff13e0fa6 100644 --- a/http/README.md +++ b/http/README.md @@ -22,6 +22,7 @@ Interest Group: [@marcopolo], [@mxinden], [@marten-seemann] - [Choosing between libp2p streams vs plain HTTPS](#choosing-between-libp2p-streams-vs-plain-https) - [Implementation recommendations](#implementation-recommendations) - [Example – Go](#example--go) +- [Future work](#future-work) - [Prior art](#prior-art) # Context @@ -42,24 +43,31 @@ as _plain https_) or on top of a libp2p stream. ## Why not two separate stacks? -Having libp2p as the abstraction over _how_ the HTTP request gets sent gives developers a lot of benefits for free, such as: +Having libp2p as the abstraction over _how_ the HTTP request gets sent gives +developers a lot of benefits for free, such as: 1. NAT traversal: You can make an HTTP request to a peer that's behind a NAT. -1. Fewer connections: If you already have a libp2p connection, we can use that to create a stream for the HTTP request. The HTTP request will be faster since you don't have to pay the two round trips to establish the connection. -1. Allows JS clients to make HTTPS requests to _any_ peer via WebTransport or WebRTC. -1. Allows more reuse of the protocol logic, just like how applications can integrate GossipSub, bitswap, graphsync, and Kademlia. +1. Fewer connections: If you already have a libp2p connection, we can use that + to create a stream for the HTTP request. The HTTP request will be faster since + you don't have to pay the two round trips to establish the connection. +1. Allows JS clients to make HTTPS requests to _any_ peer via WebTransport or + WebRTC. +1. Allows more reuse of the protocol logic, just like how applications can + integrate GossipSub, bitswap, graphsync, and Kademlia. 1. You get mutual authentication of peer IDs automatically. ## Why HTTP rather than a custom request/response protocol? -HTTP has been around for 30+ years, and it isn't going anywhere. Developers are already very familiar with it. There's is no need to reinvent the wheel here. +HTTP has been around for 30+ years, and it isn't going anywhere. Developers are +already very familiar with it. There's is no need to reinvent the wheel here. # Implementation ## HTTP over libp2p streams -If we have an existing libp2p connection that supports streams, we can run the HTTP protocol as follows: +If we have an existing libp2p connection that supports streams, we can run the +HTTP protocol as follows: Client: 1. Open a new stream to the target peer. @@ -70,7 +78,8 @@ Client: Server: 1. Register a stream handler for the `/libp2p-http` protocol. -1. On receiving a new stream speaking `/libp2p-http`, parse the HTTP request and pass it to the HTTP handler. +1. On receiving a new stream speaking `/libp2p-http`, parse the HTTP request and + pass it to the HTTP handler. 1. Write the response from the HTTP handler to the stream. 1. Close the stream when finished writing the response. @@ -78,7 +87,9 @@ Server: This is nothing more than a thin wrapper over standard HTTP. The only thing libp2p should do here is ensure that we verify the peer's TLS certificate as -defined by the [tls spec](../tls/tls.md). This SHOULD be interoperable with standard HTTP clients who pass a correct TLS cert. For example curl should work fine: +defined by the [tls spec](../tls/tls.md). This SHOULD be interoperable with +standard HTTP clients who pass a correct TLS cert. For example curl should work +fine: ``` $ curl --insecure --cert ./client.cert --key ./client.key https://127.0.0.1:9561/echo -d "Hello World" @@ -96,13 +107,15 @@ connection or a libp2p connection or expose this as an option to users. # Implementation recommendations -Each implementation should decide how this works, but the general recommendations are: +Each implementation should decide how this works, but the general +recommendations are: 1. Make this look and feel like a normal HTTP client and server. There's no -benefit of doing things differently here, and the familiarity will let people -build things faster. + benefit of doing things differently here, and the familiarity will let people + build things faster. -1. Aim to make the returned libp2p+HTTP objects interop with the general HTTP ecosystem of the language. +1. Aim to make the returned libp2p+HTTP objects interop with the general HTTP + ecosystem of the language. ## Example – Go @@ -135,6 +148,19 @@ We can create a client that accepts standard types For more details see the implementation [PR](https://github.com/libp2p/go-libp2p/pull/1874). +# Future work + +This spec aims to define HTTP as the request/response protocol abstraction for +libp2p. It also defines how libp2p+HTTP can run on top of the existing libp2p +stream abstraction _or_ a plain HTTPS connection with a certificate that +includes the libp2p extension. + +The next step is to define how to use a plain HTTPS connection with a server +certificate that doesn't have the libp2p extension (e.g. a certificate you get +from letsencrypt) in both interactive and static contexts. Some ideas are +[outlined +here](https://github.com/libp2p/specs/pull/477#issuecomment-1311988037). + # Prior art - Historical discussion: https://pl-strflt.notion.site/libp2p-HTTP-4f7d7ff3350a462a875bbfc41b3d4134