Version 2 sendcmpct for integration of segwit with CB

bip-0152.mediawiki

@@ -21,7 +21,7 @@ Thus, decreasing the bandwidth used during block relay is very useful for many i
 
 While the goal of this work is explicitly not to reduce block transfer latency, it does, as a side effect reduce block transfer latencies in some rather significant ways. Additionally, this work forms a foundation for future work explicitly targeting low-latency block transfer.
 
-==Specification==
+==Specification for version 1==
 
 ===Intended Protocol Flow===
 <img src=bip-0152/protocol-flow.png></img>
@@ -45,7 +45,7 @@ A PrefilledTransaction structure is used in HeaderAndShortIDs to provide a list
 |-
 |index||CompactSize||1, 3 bytes||Compact Size, differentially encoded since the last PrefilledTransaction in a list||The index into the block at which this transaction is
 |-
-|tx||Transaction||variable||As encoded in "tx" messages||The transaction which is in the block at index index.
+|tx||Transaction||variable||As encoded in "tx" messages sent in response to getdata MSG_TX||The transaction which is in the block at index index.
 |}
 
 ====HeaderAndShortIDs====
@@ -90,7 +90,7 @@ A BlockTransactions structure is used to provide some of the transactions in a b
 |-
 |transactions_length||CompactSize||1 or 3 bytes||As used to encode array lengths elsewhere||||The number of transactions provided
 |-
-|transactions||List of Transactions||variable||As encoded in "tx" messages||The transactions provided
+|transactions||List of Transactions||variable||As encoded in "tx" messages in response to getdata MSG_TX||The transactions provided
 |}
 
 ====Short transaction IDs====
@@ -108,9 +108,10 @@ A new inv type (MSG_CMPCT_BLOCK == 4) and several new protocol messages are adde
 # The second integer SHALL be interpreted as a little-endian version number. Nodes sending a sendcmpct message MUST currently set this value to 1.
 # Upon receipt of a "sendcmpct" message with the first and second integers set to 1, the node SHOULD announce new blocks by sending a cmpctblock message.
 # Upon receipt of a "sendcmpct" message with the first integer set to 0, the node SHOULD NOT announce new blocks by sending a cmpctblock message, but SHOULD announce new blocks by sending invs or headers, as defined by BIP130.
-# Upon receipt of a "sendcmpct" message with the second integer set to something other than 1, nodes MUST treat the peer as if they had not received the message (as it indicates the peer will provide an unexpected encoding in cmpctblock, and/or other, messages). This allows future versions to send duplicate sendcmpct messages with different versions as a part of a version handshake for future versions.
+# Upon receipt of a "sendcmpct" message with the second integer set to something other than 1, nodes MUST treat the peer as if they had not received the message (as it indicates the peer will provide an unexpected encoding in cmpctblock, and/or other, messages). This allows future versions to send duplicate sendcmpct messages with different versions as a part of a version handshake for future versions. See Protocol Versioning section, below, for more info on the specifics of the version-negotiation mechanics.
 # Nodes SHOULD check for a protocol version of >= 70014 before sending sendcmpct messages.
 # Nodes MUST NOT send a request for a MSG_CMPCT_BLOCK object to a peer before having received a sendcmpct message from that peer.
+# Nodes MUST NOT request a MSG_CMPCT_BLOCK object before having sent all sendcmpct messages to that peer which they intend to send, as the peer cannot know what version protocol to use in the response.
 
 ====MSG_CMPCT_BLOCK====
 # getdata messages may now contain requests for MSG_CMPCT_BLOCK objects.
@@ -136,7 +137,32 @@ A new inv type (MSG_CMPCT_BLOCK == 4) and several new protocol messages are adde
 ## For each short transaction ID from the original cmpctblock, in order, find the corresponding transaction either from the blocktxn message or from other sources and place it in the first available position in the block.
 # Once the block has been reconstructed, it shall be processed as normal, keeping in mind that short transaction IDs are expected to occasionally collide, and that nodes MUST NOT be penalized for such collisions, wherever they appear.
 
-===Implementation Notes===
+==Protocol Versioning==
+# The protocol version negotiation allows two nodes to agree on the versions of compact blocks which they will exchange. As it is only in a single field, it does not allow a node to support a specific version in only one direction (sending or receiving).
+# Upon connection establishment, a node SHOULD send a burst of sendcmpct messages containing every version of compact block encodings for which they are willing to support sending cmpctblock and blocktxn messages, and receiving getblocktxn messages. These messages SHOULD be ordered in the order of the priority which the node wishes to receive cmpctblock/blocktxn messages, with the highest-priority version sendcmpct message sent first.
+# The encoding version used to send a cmpctblock or blocktxn message or to receive a getblocktxn message MUST be the second integer (version number) in the first sendcmpct message received for which a sendcmpct message with the same version number was sent.
+# Nodes MUST NOT send a sendcmpct message which contains a version number other than the version number which has been negotiated for receiving cmpctblock/blocktxn messages after sending a request for a MSG_CMPCT_BLOCK object, sending a cmpctblock, getblocktxn, blocktxn, or pong message.
+# As a node must send all sendcmpct messages which contain a novel version announcement before any other compact block-related messages, it is possible to determine which version of compact blocks will be used for each object received. It is, however, not possible to know which version will be used to encode the response to a request for a compact block object before any MSG_CMPCT_BLOCK-containing getdata, cmpctblock, getblocktxn, blocktxn, or ping/pong messages have been exchanged.
+# Thus, if a node wishes to determine exactly which version of compact blocks will be used before requesting a compact block object, it must send all of its sendcmpct version announcements, followed by a ping, and wait for the pong response to ensure it has received all sendcmpctblock version announcement messages from the remote peer. Nodes can, obviously, however, determine that the version used will be at least a certain version (in their priority order) after having received a sendcmpct message from the remote peer containing that version as the second integer.
+
+===Sample Version Implementation===
+# By way of example, an implementation of the above protocol might look like the following.
+# Upon exchanging version/verack messages, a node immediately sends its list of sendcmpct announcements to the other side, with the version which it wants to receive sent first.
+# Upon receiving the first sendcmpct announcement with a protocol version which is understood from the remote peer, a node will "lock in" the compact block encoding version which will be used to encode compact blocks to that peer.
+# The node then sets the current receive-protocol-version in use on the connection as that version, and uses it to decode new compact block messages.
+# Upon receiving subsequent sendcmpct announcements with a protocol version which is understood from the remote peer, a node will check if that protocol version is higher-receive-priority than the current receive-protocol-version in use on the connection, and switch to that version for decoding new compact block messages received.
+# A node might wish to keep a flag for each peer which indicates compact block version negotiation is complete, which can be set upon receiving any compact block-related, or pong message.
+# The above implementation requires only a compile-time list of supported versions in some static priority order, two version fields per peer, and an optional negotiation-complete boolean per-peer.
+
+==Specification for version 2==
+Compact blocks version 2 is almost identical to version 1, but supports segregated witness transactions (BIP 141 and BIP 144). The changes are:
+
+# The second integer (version number) inside sendcmpct is 2 instead of 1 (see Protocol Versioning section, above).
+# Transactions inside cmpctblock messages (both those used as direct announcement and those in response to getdata) and in blocktxn should include witness data, using the same format as responses to getdata MSG_WITNESS_TX, specified in BIP144.
+# Short transaction IDs sent to us in cmpctblock messages, and sent by us in getblocktxn messages, are computed using the same process as in version 1, but using the wtxid as defined in BIP 141 instead of the txid.
+# Upon receipt of a getdata containing a request for a MSG_CMPCT_BLOCK object for which a cmpctblock message is not sent in response, the block message containing the requested block in non-compact form MUST be encoded with witnesses (as is sent in reply to a MSG_WITNESS_BLOCK getdata) if the protocol version used to encode the cmpctblock message would have been 2, and encoded without witnesses (as is sent in response to a MSG_BLOCK getdata) if the protocol version used to encode the cmpctblock message would have been 1.
+
+==Implementation Notes==
 # For nodes which have sufficient inbound bandwidth, sending a sendcmpct message with the first integer set to 1 to up to 3 peers is RECOMMENDED. If possible, it is RECOMMENDED that those peers be selected based on their past performance in providing blocks quickly (eg the three peers which provided the highest number of the recent N blocks the quickest), allowing nodes to receive blocks which come from those peers in only 0.5*RTT.
 
 # Nodes MUST NOT send such sendcmpct messages to more than three peers, as it encourages wasting outbound bandwidth across the network.
@@ -153,7 +179,7 @@ A new inv type (MSG_CMPCT_BLOCK == 4) and several new protocol messages are adde
 
 # Note that the MSG_CMPCT_BLOCK section does not require that nodes respond to MSG_CMPCT_BLOCK getdata requests for blocks which they did not recently announce. This allows nodes to calculate cmpctblock messages at announce-time instead of at request-time. Blocks which are requested with a MSG_CMPCT_BLOCK getdata, but which are not responded to with a cmpctblock message MUST be responded to with a block message, allowing nodes to request all blocks using MSG_CMPCT_BLOCK getdatas and rely on their peer to pick an appropriate response.
 
-# While the current version sends transactions with the same encodings as is used in tx messages and elsewhere in the protocol, the version field in sendcmpct is intended to allow this to change in the future. For this reason, it is recommended that the code used to decode PrefilledTransaction and BlockTransactions messages be prepared to take a different transaction encoding, if and when the version field in sendcmpct changes in a future BIP.
+# While the current version sends transactions with the same encodings as are used in tx messages and elsewhere in the protocol, the version field in sendcmpct is intended to allow this to change in the future. For this reason, it is recommended that the code used to decode PrefilledTransaction and BlockTransactions messages be prepared to take a different transaction encoding, if and when the version field in sendcmpct changes in a future BIP.
 
 # Any undefined behavior in this spec may cause failure to transfer block to, peer disconnection by, or self-destruction by the receiving node. A node receiving non-minimally-encoded CompactSize encodings should make a best-effort to eat the sender's cat.
 
@@ -190,13 +216,26 @@ In case 1, we're good. In cases 2, 3, or 4, we request the full transaction beca
 
 This means that ''B = 48'' bits short IDs suffice for blocks with up to ''t = 10000'' transactions, mempools up to ''m = 100000'' transactions, with failure to reconstruct at most one in ''F = 281474'' blocks. Since failure to reconstruct just means we fall back to normal inv/header based relay, it isn't necessary to avoid such failure completely. It just needs to be sufficiently rare they have a lower impact than random transmission failures (for example, network disconnection, node overloaded, ...).
 
+====Separate version for segregated witness====
+
+The changes to transaction and block relay in BIP 144 introduce separate MSG_FILTERED_ versions of messages in getdata,
+allowing a receiver to choose individually where witness data is wanted.
+
+This method is not useful for compact blocks because `cmpctblock` blocks can be sent unsolicitedly in high-bandwidth
+mode, so we need to negotiate at least whether those should include witness data up front. There is little use for a
+validating node that only sometimes processes witness data, so we may as well use that negotiation for everything and
+turn it into a separate protocol version. We also need a means to distinguish different versions of the same transaction
+with different witnesses for correct reconstruction, so this also forces us to use wtxids instead of txids for short IDs
+everywhere in that case.
+
 ==Backward compatibility==
 
 Older clients remain fully compatible and interoperable after this change.
 
 ==Implementation==
 
-https://github.com/bitcoin/bitcoin/pull/8068
+https://github.com/bitcoin/bitcoin/pull/8068 for version 1.
+https://github.com/bitcoin/bitcoin/pull/8393 for version 2.
 
 ==Acknowledgements==