healthcheck: make sure chain backend has enough outbound peers #8576
Conversation
WalkthroughThe update introduces a new mechanism to ensure a healthy connection to the network for Changes
Assessment against linked issues
Possibly related issues
Recent Review StatusConfiguration used: CodeRabbit UI Files selected for processing (2)
Additional comments not posted (4)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
4c0600e to
b3cbafd
Compare
c13991f to
e118b05
Compare
6d57a6a to
2a5f09e
Compare
2a5f09e to
2628df2
Compare
There was a problem hiding this comment.
Looks good! Agreed with @guggero re reducing logs for local networks 👍
Left some minor nits too
chainreg/chainregistry.go
Outdated
| err = checkOutboundPeers(chainConn) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| return nil |
There was a problem hiding this comment.
can just return checkOutboundPeers(chainConn)
chainreg/chainregistry.go
Outdated
| err = checkOutboundPeers(chainRPC.Client) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| return nil |
There was a problem hiding this comment.
return checkOutboundPeers(chainConn)
chainreg/chainregistry.go
Outdated
| } | ||
| } | ||
|
|
||
| if outboundPeers < 6 { |
There was a problem hiding this comment.
perhaps make the magic number a constant?
| @@ -263,6 +263,11 @@ bitcoin peers' feefilter values into account](https://github.com/lightningnetwor | |||
| types](https://github.com/lightningnetwork/lnd/pull/8554) defined in | |||
| `btcd/rpcclient`. | |||
|
|
|||
| `[checkOutboundPeers](https://github.com/lightningnetwork/lnd/pull/8576) is | |||
| @@ -263,6 +263,11 @@ bitcoin peers' feefilter values into account](https://github.com/lightningnetwor | |||
| types](https://github.com/lightningnetwork/lnd/pull/8554) defined in | |||
| `btcd/rpcclient`. | |||
|
|
|||
| `[checkOutboundPeers](https://github.com/lightningnetwork/lnd/pull/8576) is | |||
| added to `chainHealthCheck` to make sure chain backend `bitcoind` and `btcd` | |||
| maintains a healthy connection to the network by checking the number of | |||
There was a problem hiding this comment.
s/maintains/maintain since you are talking about multiple backends
e9016b5 to
37fc790
Compare
chainreg/chainregistry.go
Outdated
| // On local test networks we usually don't have multiple | ||
| // chain backend peers, so we can skip that test. | ||
| if cfg.Bitcoin.SimNet || cfg.Bitcoin.RegTest { | ||
| return nil | ||
| } | ||
|
|
There was a problem hiding this comment.
one last nit:
I think we should put this check in the HealthCheck call back func instead of adding it here. That way when I read the call to checkOutboundPeers then can just assume that that is what it will do & dont need to go into the method itself to go see that it will only actually do that if the network is mainnet.
There was a problem hiding this comment.
I thought the same when I put this check inside the checkOutboundPeers function. Thanks, good point!
In this commit we add `checkOutboundPeers` function to the `cc.HealthCheck` function.
37fc790 to
130fdbd
Compare
| // checkOutboundPeers checks the number of outbound peers connected to the | ||
| // provided RPC client. If the number of outbound peers is below 6, a warning | ||
| // is logged. This function is intended to ensure that the chain backend | ||
| // maintains a healthy connection to the network. | ||
| func checkOutboundPeers(client *rpcclient.Client) error { | ||
| peers, err := client.GetPeerInfo() | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| var outboundPeers int | ||
| for _, peer := range peers { | ||
| if !peer.Inbound { | ||
| outboundPeers++ | ||
| } | ||
| } | ||
|
|
||
| if outboundPeers < DefaultMinOutboundPeers { | ||
| log.Warnf("The chain backend has an insufficient number "+ | ||
| "of connected outbound peers (%d connected, expected "+ | ||
| "minimum is %d) which can be a security issue. "+ | ||
| "Connect to more trusted nodes manually if necessary.", | ||
| outboundPeers, DefaultMinOutboundPeers) | ||
| } | ||
|
|
||
| return nil | ||
| } |
There was a problem hiding this comment.
The checkOutboundPeers function effectively logs a warning if the number of outbound peers is below the defined threshold. This function is well-implemented, but consider enhancing it by returning a specific error when the number of outbound peers is critically low, allowing for more decisive action than just logging.
if outboundPeers < criticalThreshold {
- log.Warnf("...")
+ return fmt.Errorf("critical low number of outbound peers: %d", outboundPeers)
}Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| // checkOutboundPeers checks the number of outbound peers connected to the | |
| // provided RPC client. If the number of outbound peers is below 6, a warning | |
| // is logged. This function is intended to ensure that the chain backend | |
| // maintains a healthy connection to the network. | |
| func checkOutboundPeers(client *rpcclient.Client) error { | |
| peers, err := client.GetPeerInfo() | |
| if err != nil { | |
| return err | |
| } | |
| var outboundPeers int | |
| for _, peer := range peers { | |
| if !peer.Inbound { | |
| outboundPeers++ | |
| } | |
| } | |
| if outboundPeers < DefaultMinOutboundPeers { | |
| log.Warnf("The chain backend has an insufficient number "+ | |
| "of connected outbound peers (%d connected, expected "+ | |
| "minimum is %d) which can be a security issue. "+ | |
| "Connect to more trusted nodes manually if necessary.", | |
| outboundPeers, DefaultMinOutboundPeers) | |
| } | |
| return nil | |
| } | |
| // checkOutboundPeers checks the number of outbound peers connected to the | |
| // provided RPC client. If the number of outbound peers is below 6, a warning | |
| // is logged. This function is intended to ensure that the chain backend | |
| // maintains a healthy connection to the network. | |
| func checkOutboundPeers(client *rpcclient.Client) error { | |
| peers, err := client.GetPeerInfo() | |
| if err != nil { | |
| return err | |
| } | |
| var outboundPeers int | |
| for _, peer := range peers { | |
| if !peer.Inbound { | |
| outboundPeers++ | |
| } | |
| } | |
| if outboundPeers < DefaultMinOutboundPeers { | |
| log.Warnf("The chain backend has an insufficient number "+ | |
| "of connected outbound peers (%d connected, expected "+ | |
| "minimum is %d) which can be a security issue. "+ | |
| "Connect to more trusted nodes manually if necessary.", | |
| outboundPeers, DefaultMinOutboundPeers) | |
| } | |
| if outboundPeers < criticalThreshold { | |
| return fmt.Errorf("critical low number of outbound peers: %d", outboundPeers) | |
| } | |
| return nil | |
| } |
There was a problem hiding this comment.
consider enhancing it by returning a specific error when the number of outbound peers is critically low, allowing for more decisive action than just logging.
I don't think this is what we want since bitcoind should recover from this automatically.
There was a problem hiding this comment.
@mohamedawnallah, thank you for your input. It's a valid point that bitcoind is designed to manage its peer connections autonomously and should be capable of recovering from a low number of outbound peers on its own. Given this, it makes sense to rely on bitcoind's built-in mechanisms for maintaining a healthy connection to the network without introducing additional complexity into lnd. Your decision to keep the current implementation as is, focusing on logging for awareness while trusting in bitcoind's recovery capabilities, is well-understood.
|
@guggero Is it ready to be merged? |
Change Description
Closes #8487
Steps to Test
Steps for reviewers to follow to test the change.
Pull Request Checklist
Testing
Code Style and Documentation
[skip ci]in the commit message for small changes.📝 Please see our Contribution Guidelines for further guidance.