Security: Running Zebra nodes should eventually stop trying to contact peers that always fail

[teor2345]

Motivation

Zebra will keep trying individual Failed peers, even if they have never succeeded. This is a distributed denial of service risk, and it places extra load on the network.

Scheduling

We should fix this issue before NU5 mainnet activation, so this bug doesn't cause a denial of service from old Zebra versions when NU6 activates.

Suggestions

This fix depends on #1849, #1867, and #1871.

Zebra should stop trying to contact peers that haven't had a successful connection for 3 days. We've chosen this time to allow admins to restart their nodes after a weekend failure. (We might want to change this to a longer timeframe in a future upgrade, once Zebra is stable.)

Zebra should delete peers from the AddressBook where the:

• last_success_time is older than 3 days
• last_success_time is None, and the untrusted_last_seen is older than 3 days (requires the far-future fix in Security: Zebra should stop believing far-future last_seen times from peers #1871)
• this task should be triggered by an interval timer that runs every minute, see crawl_and_dial for an example of this kind of address book task
  • the task should directly run an address book method containing these checks, to make testing easier
  • the task should also be run if the address book gets too large, but we'll do that in Security: Zebra's address book can use all available memory #1873

Zebra should also:

• provide an accessor method on MetaAddr to simplify the interface to these times

To avoid sending old peers to other nodes, Zebra should:

• keep the earliest last seen times gossiped by other peers
• avoid sending unreachable DNS seeder addresses to peers
  • these addresses don't have any times, so we have no way of removing them from the network
• before sending unreachable gossiped addresses, subtract 30 minutes from their last seen times
  • this effectively becomes a time-to-live for unreachable addresses, giving them 6 gossips before Zebra stops forwarding them (based on the 3 hour gossip limit from Security: Zebra should stop gossiping unreachable addresses to other nodes, Action: re-deploy all nodes #1867)
  • the Bitcoin reference says we shouldn't change these times, but we already sanitize them

To avoid accepting old peers from other nodes, Zebra should:

• ignore peers that are older than 3 days
  • Zebra should count back 3 days from the newest peer timestamp sent by the other peer, to compensate for clock skew (otherwise, different local and remote clocks could make us delete all the peers)
  • For details, see the clock skew adjustment algorithm in Security: Zebra should stop believing far-future last_seen times from peers #1871

Property testing:

• Generate some random MetaAddrs
• Put them in an AddressBook
• Run the deletion method on the address book
• Make sure that old last_success_times and untrusted_last_seen_times are handled correctly

For testing:

• add a debug_peer_deletion_age config that sets the deletion age and interval timer
• add an acceptance test that sets it to 15 seconds
• add a log when all the peers are deleted, and check for that log - "hint: upgrade or restart Zebra"

Performance Analysis

If we check for old peers every time a new peer is requested, we could spend a lot of time checking for deletions. Instead, we should scan the address book at regular intervals in a new task.

Alternatives

This is a critical security issue, so we must do something.

We could keep a failure count for each peer. This design has usability issues on unreliable networks, because all the peers can fail at the same time. (Our reconnection rate limit and peer deletion timeout already limit us to ~2000 failed connections per peer over 3 days.)

We could filter peers whenever the AddressBook is accessed, rather than deleting them. But this is tricky - it might need interior mutability. It would also require an abstraction layer, to make sure we intercept all accesses.

We could avoid deleting peers in NeverAttempted... states, so that we try each peer at least once before deleting it. But this would risk a memory DoS, if we get a lot of gossiped peers.

Context

zcashd does not have this issue.

zcashd has a support interval of around 16 weeks between required upgrades, with new versions coming out every 6 weeks.

Follow-Up Tasks

This fix doesn't deal with denial of service from nodes that are regularly restarted, that's #1870.

[teor2345]

This fix is required for NU5 mainnet activation. If the issue happens on testnet, we should be able to work around it.