Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

healthcheck: make sure chain backend has enough outbound peers #8576

Merged
merged 2 commits into from
Apr 10, 2024
Merged

healthcheck: make sure chain backend has enough outbound peers #8576

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
df9f148
chainreg: add `checkOutboundPeers` function
mohamedawnallah Mar 31, 2024
130fdbd
docs: update the release-notes-0.18.0.md
mohamedawnallah Apr 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 63 additions & 2 deletions chainreg/chainregistry.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,11 @@ const (
// DefaultBitcoinStaticMinRelayFeeRate is the min relay fee used for
// static estimators.
DefaultBitcoinStaticMinRelayFeeRate = chainfee.FeePerKwFloor

// DefaultMinOutboundPeers is the min number of connected
// outbound peers the chain backend should have to maintain a
// healthy connection to the network.
DefaultMinOutboundPeers = 6
)

// PartialChainControl contains all the primary interfaces of the chain control
Expand Down Expand Up @@ -504,7 +509,21 @@ func NewPartialChainControl(cfg *Config) (*PartialChainControl, func(), error) {

cc.HealthCheck = func() error {
_, err := chainConn.RawRequest(cmd, nil)
return err
if err != nil {
return err
}

// On local test networks we usually don't have multiple
// chain backend peers, so we can skip
// the checkOutboundPeers test.
if cfg.Bitcoin.SimNet || cfg.Bitcoin.RegTest {
return nil
}

// Make sure the bitcoind chain backend maintains a
// healthy connection to the network by checking the
// number of outbound peers.
return checkOutboundPeers(chainConn)
}

case "btcd":
Expand Down Expand Up @@ -613,7 +632,21 @@ func NewPartialChainControl(cfg *Config) (*PartialChainControl, func(), error) {
// Use a query for our best block as a health check.
cc.HealthCheck = func() error {
_, _, err := cc.ChainSource.GetBestBlock()
return err
if err != nil {
return err
}

guggero marked this conversation as resolved.
Show resolved Hide resolved
// On local test networks we usually don't have multiple
// chain backend peers, so we can skip
// the checkOutboundPeers test.
if cfg.Bitcoin.SimNet || cfg.Bitcoin.RegTest {
return nil
}

// Make sure the btcd chain backend maintains a
// healthy connection to the network by checking the
// number of outbound peers.
return checkOutboundPeers(chainRPC.Client)
}

// If we're not in simnet or regtest mode, then we'll attempt
Expand Down Expand Up @@ -840,3 +873,31 @@ var (
},
}
)

// checkOutboundPeers checks the number of outbound peers connected to the
// provided RPC client. If the number of outbound peers is below 6, a warning
// is logged. This function is intended to ensure that the chain backend
// maintains a healthy connection to the network.
func checkOutboundPeers(client *rpcclient.Client) error {
peers, err := client.GetPeerInfo()
if err != nil {
return err
}

var outboundPeers int
for _, peer := range peers {
if !peer.Inbound {
outboundPeers++
}
}

if outboundPeers < DefaultMinOutboundPeers {
log.Warnf("The chain backend has an insufficient number "+
"of connected outbound peers (%d connected, expected "+
"minimum is %d) which can be a security issue. "+
"Connect to more trusted nodes manually if necessary.",
outboundPeers, DefaultMinOutboundPeers)
}

return nil
}
Comment on lines +877 to +903
Copy link
Contributor

@coderabbitai coderabbitai bot Apr 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The checkOutboundPeers function effectively logs a warning if the number of outbound peers is below the defined threshold. This function is well-implemented, but consider enhancing it by returning a specific error when the number of outbound peers is critically low, allowing for more decisive action than just logging.

if outboundPeers < criticalThreshold {
-   log.Warnf("...")
+   return fmt.Errorf("critical low number of outbound peers: %d", outboundPeers)
}
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
// checkOutboundPeers checks the number of outbound peers connected to the
// provided RPC client. If the number of outbound peers is below 6, a warning
// is logged. This function is intended to ensure that the chain backend
// maintains a healthy connection to the network.
func checkOutboundPeers(client *rpcclient.Client) error {
peers, err := client.GetPeerInfo()
if err != nil {
return err
}
var outboundPeers int
for _, peer := range peers {
if !peer.Inbound {
outboundPeers++
}
}
if outboundPeers < DefaultMinOutboundPeers {
log.Warnf("The chain backend has an insufficient number "+
"of connected outbound peers (%d connected, expected "+
"minimum is %d) which can be a security issue. "+
"Connect to more trusted nodes manually if necessary.",
outboundPeers, DefaultMinOutboundPeers)
}
return nil
}
// checkOutboundPeers checks the number of outbound peers connected to the
// provided RPC client. If the number of outbound peers is below 6, a warning
// is logged. This function is intended to ensure that the chain backend
// maintains a healthy connection to the network.
func checkOutboundPeers(client *rpcclient.Client) error {
peers, err := client.GetPeerInfo()
if err != nil {
return err
}
var outboundPeers int
for _, peer := range peers {
if !peer.Inbound {
outboundPeers++
}
}
if outboundPeers < DefaultMinOutboundPeers {
log.Warnf("The chain backend has an insufficient number "+
"of connected outbound peers (%d connected, expected "+
"minimum is %d) which can be a security issue. "+
"Connect to more trusted nodes manually if necessary.",
outboundPeers, DefaultMinOutboundPeers)
}
if outboundPeers < criticalThreshold {
return fmt.Errorf("critical low number of outbound peers: %d", outboundPeers)
}
return nil
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider enhancing it by returning a specific error when the number of outbound peers is critically low, allowing for more decisive action than just logging.

I don't think this is what we want since bitcoind should recover from this automatically.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mohamedawnallah, thank you for your input. It's a valid point that bitcoind is designed to manage its peer connections autonomously and should be capable of recovering from a low number of outbound peers on its own. Given this, it makes sense to rely on bitcoind's built-in mechanisms for maintaining a healthy connection to the network without introducing additional complexity into lnd. Your decision to keep the current implementation as is, focusing on logging for awareness while trusting in bitcoind's recovery capabilities, is well-understood.

5 changes: 5 additions & 0 deletions docs/release-notes/release-notes-0.18.0.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,11 @@ bitcoin peers' feefilter values into account](https://github.com/lightningnetwor
types](https://github.com/lightningnetwork/lnd/pull/8554) defined in
`btcd/rpcclient`.

* [checkOutboundPeers](https://github.com/lightningnetwork/lnd/pull/8576) is
added to `chainHealthCheck` to make sure chain backend `bitcoind` and `btcd`
maintain a healthy connection to the network by checking the number of
outbound peers if they are below 6.

### Logging
* [Add the htlc amount](https://github.com/lightningnetwork/lnd/pull/8156) to
contract court logs in case of timed-out HTLCs in order to easily spot dust
Expand Down
Loading