Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exit on startup: unable to force close channel #3477

Closed
mcgotts opened this issue Sep 6, 2019 · 6 comments
Closed

Exit on startup: unable to force close channel #3477

mcgotts opened this issue Sep 6, 2019 · 6 comments

Comments

@mcgotts
Copy link

mcgotts commented Sep 6, 2019

Background

Attempted to run "lncli feereport" on node yesterday and ran into an error where it said "rpc error: code = Unknown desc = no policy for outgoing channel x". Could not find the channel in "lncli listchannels" but checking "lncli getchaninfo" I saw the channel with both policies as "null". I then tried to force close and got the response "channel is already being closed". Running "lncli pendingchannels" I saw the channel in waiting to be closed.

Thinking a reboot might help the situation, I restarted both lnd and bitcoind however upon restart, lnd starts up but then crashes with the error: "Unable to start server: -26: bad-txns-in-belowout, value in (x) < value out (x) (code 16)".

Every time I restart the node I get this same error now so can't even get into my node to try to replicate the original "lncli feereport" error.

Your environment

lnd: lnd version 0.7.0-beta commit=v0.7.0-beta-93-g377b7bf3ce493690fadc9d7be062062b006074c9
OS: Linux 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Bitcoind: Bitcoin Core Daemon version v0.17.1.0-gef70f9b52b851c7997a9f1a0834714e3eebc1fd8

Steps to reproduce

Starting LND (using supervisor - sudo supervisorctl start lnd)

Expected behaviour

LND should start normally

Actual behaviour

LND starts but shortly after I unlock it crashes, requiring unlock again.

Logs:

2019-09-06 15:24:23.858 [ERR] CNCT: ChannelArbitrator(x:1): unable to broadcast close tx: -26: bad-txns-in-belowout, value in (x) < value out (x) (code 16)
2019-09-06 15:24:23.858 [ERR] CNCT: ChannelArbitrator(x:1): unable to advance state: -26: bad-txns-in-belowout, value in (x) < value out (x) (code 16)
2019-09-06 15:24:23.863 [ERR] CNCT: ChannelArbitrator(x:0): unable to progress resolver: quitting
2019-09-06 15:24:23.866 [ERR] LTND: Unable to start server: -26: bad-txns-in-belowout, value in (x) < value out (x) (code 16)
2019-09-06 15:24:23.866 [INF] RPCS: Stopping RPC Server
2019-09-06 15:24:24.941 [INF] RPCS: password gRPC proxy started at 0.0.0.0:8080
2019-09-06 15:24:24.941 [INF] RPCS: password RPC server listening on 127.0.0.1:10009
2019-09-06 15:25:50.431 [INF] LTND: Received terminated
2019-09-06 15:25:50.432 [ERR] LTND: Unable to set up wallet password listeners: shutting down

@Roasbeef Roasbeef changed the title Exit on startup: Unable to start server: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16) Exit on startup: unable to force close channel Sep 7, 2019
@Roasbeef
Copy link
Member

Roasbeef commented Sep 7, 2019

Looks similar to #1267 (a very old issue). How long has this node been in operation?

If you apply this patch, then you'll be able to start up again:

diff --git a/contractcourt/channel_arbitrator.go b/contractcourt/channel_arbitrator.go
index b90707686..030eccb43 100644
--- a/contractcourt/channel_arbitrator.go
+++ b/contractcourt/channel_arbitrator.go
@@ -403,7 +403,7 @@ func (c *ChannelArbitrator) Start() error {
 
 		default:
 			c.cfg.BlockEpochs.Cancel()
-			return err
+			//return err
 		}
 	}

Looks like the you don't have any coins in the channel, so you can use the abandonchannel command to forget the channel.

note added: abandonchannel is a dangerous command and can lead to loss of funds if used incorrectly - ask in the LND slack channel if unsure

Also while you're at it, I'd recommend updating to v0.7.1-beta since you're on a dirty (has commits mid-way between two versions)v0.7.0 branch that likely contains some behavior that was fixed during the rc cycle.

(edited your initial issue to remove some possibly sensitive content)

@mcgotts
Copy link
Author

mcgotts commented Sep 9, 2019

Thanks roasbeef - will try to add the patch this morning and see if I can get back up and running and will then attempt to use abandonchannel. Node has been operation for about 7 months.

Will update once I'm able to test.

@McGiggity
Copy link

Still appear to be having the same issue after updating to 0.7.1 and applying the patch (commented out return err from contract court). Still crashing on startup after unlocking node.

Debug error logs as follows (let me know if it would be helpful to see the full log of startup):

2019-09-06 15:24:23.858 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to broadcast close tx: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-06 15:24:23.858 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to advance state: -26: bad-txns-in-belowout, value in (0.00057) < valu e out (0.14996174) (code 16)
2019-09-06 15:24:23.863 [ERR] CNCT: ChannelArbitrator(8f40b5ad71c48dc1f7ebdf255089764ed2251c3fee044985b2e90a960c57fc8b:0): unable to progress resolver: quitting
2019-09-06 15:24:23.866 [ERR] LTND: Unable to start server: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-06 15:25:50.432 [ERR] LTND: Unable to set up wallet password listeners: shutting down
2019-09-09 14:06:28.682 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to broadcast close tx: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 14:06:28.682 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to advance state: -26: bad-txns-in-belowout, value in (0.00057) < valu e out (0.14996174) (code 16)
2019-09-09 14:06:28.688 [ERR] CNCT: ChannelArbitrator(8f40b5ad71c48dc1f7ebdf255089764ed2251c3fee044985b2e90a960c57fc8b:0): unable to progress resolver: quitting
2019-09-09 14:06:28.692 [ERR] CNCT: ChannelArbitrator(b3bcec82c25b63db9eb45cabc4c9577553e31e9f011cc3393d27267d6980dc93:1): unable to progress resolver: quitting
2019-09-09 14:06:28.692 [ERR] LTND: Unable to start server: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 14:56:14.692 [ERR] LTND: Unable to set up wallet password listeners: shutting down
2019-09-09 19:36:55.495 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to broadcast close tx: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 19:36:55.495 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to advance state: -26: bad-txns-in-belowout, value in (0.00057) < valu e out (0.14996174) (code 16)
2019-09-09 19:36:55.501 [ERR] CNCT: ChannelArbitrator(8f40b5ad71c48dc1f7ebdf255089764ed2251c3fee044985b2e90a960c57fc8b:0): unable to progress resolver: quitting
2019-09-09 19:36:55.502 [ERR] LTND: Unable to start server: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 19:38:32.874 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to broadcast close tx: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 19:38:32.874 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to advance state: -26: bad-txns-in-belowout, value in (0.00057) < valu e out (0.14996174) (code 16)
2019-09-09 19:38:32.882 [ERR] LTND: Unable to start server: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 19:39:31.820 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to broadcast close tx: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 19:39:31.820 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to advance state: -26: bad-txns-in-belowout, value in (0.00057) < valu e out (0.14996174) (code 16)
2019-09-09 19:39:31.832 [ERR] CNCT: ChannelArbitrator(b3bcec82c25b63db9eb45cabc4c9577553e31e9f011cc3393d27267d6980dc93:1): unable to progress resolver: quitting
2019-09-09 19:39:31.833 [ERR] CNCT: ChannelArbitrator(8f40b5ad71c48dc1f7ebdf255089764ed2251c3fee044985b2e90a960c57fc8b:0): unable to progress resolver: quitting
2019-09-09 19:39:31.836 [ERR] LTND: Unable to start server: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 19:50:14.771 [ERR] LTND: Unable to set up wallet password listeners: shutting down
2019-09-09 20:13:53.479 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to broadcast close tx: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 20:13:53.480 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to advance state: -26: bad-txns-in-belowout, value in (0.00057) < valu e out (0.14996174) (code 16)
2019-09-09 20:13:53.486 [ERR] LTND: Unable to start server: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 20:17:59.123 [ERR] LTND: Unable to set up wallet password listeners: shutting down
2019-09-09 20:18:43.200 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to broadcast close tx: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)
2019-09-09 20:18:43.200 [ERR] CNCT: ChannelArbitrator(800367ed55bbba13ba2586765ed8d681dfa5eba91602c66a2c183d676326b256:1): unable to advance state: -26: bad-txns-in-belowout, value in (0.00057) < valu e out (0.14996174) (code 16)
2019-09-09 20:18:43.203 [ERR] CNCT: ChannelArbitrator(b3bcec82c25b63db9eb45cabc4c9577553e31e9f011cc3393d27267d6980dc93:1): unable to progress resolver: quitting
2019-09-09 20:18:43.207 [ERR] LTND: Unable to start server: -26: bad-txns-in-belowout, value in (0.00057) < value out (0.14996174) (code 16)

@Roasbeef
Copy link
Member

Actually here's the proper patch to be able to restart:

diff --git a/contractcourt/chain_arbitrator.go b/contractcourt/chain_arbitrator.go
index b3cd497d2..f57ee63ad 100644
--- a/contractcourt/chain_arbitrator.go
+++ b/contractcourt/chain_arbitrator.go
@@ -511,7 +511,7 @@ func (c *ChainArbitrator) Start() error {
 	// can carry out their duties.
 	for _, arbitrator := range c.activeChannels {
 		if err := arbitrator.Start(); err != nil {
-			c.Stop()
+			//c.Stop()
 			return err
 		}
 	}

@mcgotts
Copy link
Author

mcgotts commented Sep 10, 2019

Worked further with roasbeef to get the issue of node-startup solved with the patch included at the bottom.

As a next step, used abandon channel to attempt to get rid of the zombie channel, which returned without error however still encountering the same errors when running lncli feereport ([lncli] rpc error: code = Unknown desc = no policy for outgoing channel 652233596219424769).

Appears to be related to issue #2911

Patch used to get node to start-up successfully:
image

@wpaulino
Copy link
Contributor

As a next step, used abandon channel to attempt to get rid of the zombie channel, which returned without error however still encountering the same errors when running lncli feereport ([lncli] rpc error: code = Unknown desc = no policy for outgoing channel 652233596219424769).

That seems to be an issue with the abandonchannel command itself. Could you open another issue for that instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants