Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BOLT7: extend channel range queries with optional fields #557

Merged
merged 11 commits into from
Sep 16, 2019
Merged

BOLT7: extend channel range queries with optional fields #557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
57586c0
BOLT7: query_short_channel_ids: allow requesting specific information…
sstone Jul 17, 2019
eecc7f1
BOLT7: query_channel_range: allow requesting timestamps & checksums (…
sstone Jul 17, 2019
3564370
BOLT 7: fix up TLV formatting for modern parser.
rustyrussell Jul 17, 2019
7ca9cc7
BOLT 7: fixup: add missing encoding_type in checksums_tlv.
rustyrussell Jul 17, 2019
bb6ecab
BOLT 7: fixup: specify exactly which version of CRC32.
rustyrussell Jul 17, 2019
c5aacc3
Spelling fixes
sstone Jul 22, 2019
c11c35a
Fixup: address review comments
sstone Jul 25, 2019
1a0e729
TLV Formatting changes
sstone Aug 6, 2019
927c96d
Define a feature bit for extended gossip queries
sstone Aug 6, 2019
e52e8ec
Add test vector for extended queries
sstone Aug 6, 2019
3d43748
BOTL 7: fixup: clarify when to reply with a `node_announcement`
sstone Aug 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .aspell.en.pws
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
personal_ws-1.1 en 264
bitfields
checksums
timestamps
tlv
tlvs
subtype
TLV
py
vsprintf
glibc
Expand Down Expand Up @@ -336,6 +343,7 @@ zlib
ZLIB
APIs
duplicative
CRC
DoS
ECDSA
TLV
Expand Down
120 changes: 115 additions & 5 deletions 07-routing-gossip.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -574,19 +574,47 @@ Encoding types:
* `0`: uncompressed array of `short_channel_id` types, in ascending order.
* `1`: array of `short_channel_id` types, in ascending order, compressed with zlib deflate<sup>[1](#reference-1)</sup>

This encoding is also used for arrays of other types (timestamps, flags, ...), and specified with an `encoded_` prefix. For example, `encoded_timestamps` is an array of timestamps than can be either compressed (with a `1` prefix) or uncompressed (with a `0` prefix).

Note that a 65535-byte zlib message can decompress into 67632120
bytes<sup>[2](#reference-2)</sup>, but since the only valid contents
are unique 8-byte values, no more than 14 bytes can be duplicated
across the stream: as each duplicate takes at least 2 bits, no valid
contents could decompress to more then 3669960 bytes.

Query messages can be extended with optional fields that can help reduce the number of messages needed to synchronize routing tables by enabling:

- timestamp-based filtering of `channel_update` messages: only ask for `channel_update` messages that are newer than the ones you already have.
- checksum-based filtering of `channel_update` messages: only ask for `channel_update` messages that carry different information from the ones you already have.

Nodes can signal that they support extended gossip queries with the `gossip_queries_ex` feature bit.

### The `query_short_channel_ids`/`reply_short_channel_ids_end` Messages

1. type: 261 (`query_short_channel_ids`) (`gossip_queries`)
2. data:
* [`chain_hash`:`chain_hash`]
* [`u16`:`len`]
* [`len*byte`:`encoded_short_ids`]
* [`query_short_channel_ids_tlvs`:`tlvs`]

1. tlvs: `query_short_channel_ids_tlvs`
2. types:
1. type: 1 (`query_flags`)
2. data:
* [`...*byte`:`encoded_query_flags`]

`encoded_query_flags` is an array of bitfields, one varint per bitfield, one bitfield for each `short_channel_id`. Bits have the following meaning:

| Bit Position | Meaning |
| ------------- | ---------------------------------------- |
| 0 | Sender wants `channel_announcement` |
| 1 | Sender wants `channel_update` for node 1 |
| 2 | Sender wants `channel_update` for node 2 |
| 3 | Sender wants `node_announcement` for node 1 |
| 4 | Sender wants `node_announcement` for node 2 |

Query flags must be minimally encoded, which means that one flag will be encoded with a single byte.

1. type: 262 (`reply_short_channel_ids_end`) (`gossip_queries`)
2. data:
Expand All @@ -611,6 +639,10 @@ The sender:
- MAY send this if it receives a `channel_update` for a
`short_channel_id` for which it has no `channel_announcement`.
- SHOULD NOT send this if the channel referred to is not an unspent output.
- MAY include an optional `query_flags`. If so:
- MUST set `encoding_type`, as for `encoded_short_ids`.
- Each query flag is a minimally-encoded varint.
- MUST encode one query flag per `short_channel_id`.

The receiver:
- if the first byte of `encoded_short_ids` is not a known encoding type:
Expand All @@ -619,11 +651,31 @@ The receiver:
- MAY fail the connection.
- if it has not sent `reply_short_channel_ids_end` to a previously received `query_short_channel_ids` from this sender:
- MAY fail the connection.
- MUST respond to each known `short_channel_id` with a `channel_announcement`
and the latest `channel_update` for each end
- if the incoming message includes `query_short_channel_ids_tlvs`:
- if `encoding_type` is not a known encoding type:
- MAY fail the connection
- if `encoded_query_flags` does not decode to exactly one flag per `short_channel_id`:
- MAY fail the connection.
- MUST respond to each known `short_channel_id`:
- if the incoming message does not include `encoded_query_flags`:
- with a `channel_announcement` and the latest `channel_update` for each end
- MUST follow with any `node_announcement`s for each `channel_announcement`
- otherwise:
- We define `query_flag` for the Nth `short_channel_id` in
`encoded_short_ids` to be the Nth varint of the decoded
`encoded_query_flags`.
- if bit 0 of `query_flag` is set:
- MUST reply with a `channel_announcement`
- if bit 1 of `query_flag` is set and it has received a `channel_update` from `node_id_1`:
- MUST reply with the latest `channel_update` for `node_id_1`
- if bit 2 of `query_flag` is set and it has received a `channel_update` from `node_id_2`:
sstone marked this conversation as resolved.
Show resolved Hide resolved
- MUST reply with the latest `channel_update` for `node_id_2`
cfromknecht marked this conversation as resolved.
Show resolved Hide resolved
- if bit 3 of `query_flag` is set and it has received a `node_announcement` from `node_id_1`:
- MUST reply with the latest `node_announcement` for `node_id_1`
- if bit 4 of `query_flag` is set and it has received a `node_announcement` from `node_id_2`:
- MUST reply with the latest `node_announcement` for `node_id_2`
- SHOULD NOT wait for the next outgoing gossip flush to send these.
- MUST follow with any `node_announcement`s for each `channel_announcement`
- SHOULD avoid sending duplicate `node_announcements` in response to a single `query_short_channel_ids`.
- SHOULD avoid sending duplicate `node_announcements` in response to a single `query_short_channel_ids`.
- MUST follow these responses with `reply_short_channel_ids_end`.
- if does not maintain up-to-date channel information for `chain_hash`:
- MUST set `complete` to 0.
Expand All @@ -648,6 +700,22 @@ timeouts. It also causes a natural ratelimiting of queries.
* [`chain_hash`:`chain_hash`]
* [`u32`:`first_blocknum`]
* [`u32`:`number_of_blocks`]
* [`query_channel_range_tlvs`:`tlvs`]

1. tlvs: `query_channel_range_tlvs`
2. types:
1. type: 1 (`query_option`)
2. data:
* [`varint`:`query_option_flags`]

`query_option_flags` is a bitfield represented as a minimally-encoded varint. Bits have the following meaning:

| Bit Position | Meaning |
| ------------- | ----------------------- |
| 0 | Sender wants timestamps |
| 1 | Sender wants checksums |

Though it is possible, it would not be very useful to ask for checksums without asking for timestamps too: the receiving node may have an older `channel_update` with a different checksum, asking for it would be useless. And if a `channel_update` checksum is actually 0 (which is quite unlikely) it will not be queried.

1. type: 264 (`reply_channel_range`) (`gossip_queries`)
2. data:
Expand All @@ -657,8 +725,42 @@ timeouts. It also causes a natural ratelimiting of queries.
* [`byte`:`complete`]
* [`u16`:`len`]
* [`len*byte`:`encoded_short_ids`]
* [`reply_channel_range_tlvs`:`tlvs`]

This allows a query for channels within specific blocks.
1. tlvs: `query_channel_range_tlvs`
2. types:
1. type: 1 (`timestamps_tlv`)
2. data:
* [`...*byte`:`encoded_timestamps`]
1. type: 3 (`checksums_tlv`)
2. data:
* [`...*byte`:`checksums`]

For a single `channel_update`, timestamps are encoded as:

1. subtype: `channel_update_timestamps`
2. data:
* [`u32`:`timestamp_node_id_1`]
* [`u32`:`timestamp_node_id_2`]

Where:
* `timestamp_node_id_1` is the timestamp of the `channel_update` for `node_id_1`, or 0 if there was no `channel_update` from that node.
* `timestamp_node_id_2` is the timestamp of the `channel_update` for `node_id_2`, or 0 if there was no `channel_update` from that node.

For a single `channel_update`, checksums are encoded as:

1. subtype: `channel_update_checksums`
2. data:
* [`u32`:`checksum_node_id_1`]
* [`u32`:`checksum_node_id_2`]

Where:
* `checksum_node_id_1` is the checksum of the `channel_update` for `node_id_1`, or 0 if there was no `channel_update` from that node.
sstone marked this conversation as resolved.
Show resolved Hide resolved
* `checksum_node_id_2` is the checksum of the `channel_update` for `node_id_2`, or 0 if there was no `channel_update` from that node.

The checksum of a `channel_update` is the CRC32C checksum as specified in [RFC3720](https://tools.ietf.org/html/rfc3720#appendix-B.4) of this `channel_update` without its `signature` and `timestamp` fields.

This allows to query for channels within specific blocks.

#### Requirements

Expand All @@ -668,6 +770,7 @@ The sender of `query_channel_range`:
that it wants the `reply_channel_range` to refer to
- MUST set `first_blocknum` to the first block it wants to know channels for
- MUST set `number_of_blocks` to 1 or greater.
- MAY append an additional `query_channel_range_tlv`, which specifies the type of extended information it would like to receive.

The receiver of `query_channel_range`:
- if it has not sent all `reply_channel_range` to a previously received `query_channel_range` from this sender:
Expand All @@ -685,12 +788,19 @@ The receiver of `query_channel_range`:
- otherwise:
- SHOULD set `complete` to 1.

If the incoming message includes `query_option`, the receiver MAY append additional information to its reply:
- if bit 0 in `query_option_flags` is set, the receiver MAY append a `timestamps_tlv` that contains `channel_update` timestamps for all `short_chanel_id`s in `encoded_short_ids`
- if bit 1 in `query_option_flags` is set, the receiver MAY append a `checksums_tlv` that contains `channel_update` checksums for all `short_chanel_id`s in `encoded_short_ids`


#### Rationale

A single response might be too large for a single packet, and also a peer can
store canned results for (say) 1000-block ranges, and simply offer each reply
which overlaps the ranges of the request.

The addition of timestamp and checksum fields allow a peer to omit querying for redundant updates.

### The `gossip_timestamp_filter` Message

1. type: 265 (`gossip_timestamp_filter`) (`gossip_queries`)
Expand Down
14 changes: 7 additions & 7 deletions 09-features.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,13 @@ see [BOLT #1: The `init` Message](01-messaging.md#the-init-message).

These flags may only be used in the `init` message:

| Bits | Name | Description | Link |
|------|----------------------------------|---------------------------------------------------------------------------|------------------------------|
| 0/1 | `option_data_loss_protect` | Requires or supports extra `channel_reestablish` fields | [BOLT #2][bolt02-retransmit] |
| 3 | `initial_routing_sync` | Indicates that the sending node needs a complete routing information dump | [BOLT #7][bolt07-sync] |
| 4/5 | `option_upfront_shutdown_script` | Commits to a shutdown scriptpubkey when opening channel | [BOLT #2][bolt02-open] |
| 6/7 | `gossip_queries` | More sophisticated gossip control | [BOLT #7][bolt07-query] |
| Bits | Name | Description | Link |
|-------|----------------------------------|---------------------------------------------------------------------------|------------------------------|
| 0/1 | `option_data_loss_protect` | Requires or supports extra `channel_reestablish` fields | [BOLT #2][bolt02-retransmit] |
| 3 | `initial_routing_sync` | Indicates that the sending node needs a complete routing information dump | [BOLT #7][bolt07-sync] |
| 4/5 | `option_upfront_shutdown_script` | Commits to a shutdown scriptpubkey when opening channel | [BOLT #2][bolt02-open] |
| 6/7 | `gossip_queries` | More sophisticated gossip control | [BOLT #7][bolt07-query] |
| 10/11 | `gossip_queries_ex` | Gossip queries can include additional information | [BOLT #7][bolt07-query] |

## Assigned `globalfeatures` flags

Expand All @@ -35,7 +36,6 @@ The following `globalfeatures` bits are currently assigned by this specification
|------|-------------------|--------------------------------------------------------------------|---------------------------------------|
| 8/9 | `var_onion_optin` | This node requires/supports variable-length routing onion payloads | [Routing Onion Specification][bolt04] |


## Requirements

The requirements for receiving specific bits are defined in the linked sections in the table above.
Expand Down
Loading