Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't remove remote watchtower that is reporting error "StateUpdateCodeSeqNumOutOfOrder in StateUpdateReply for seqnum=1021". #6397

Closed
pcwd opened this issue Apr 8, 2022 · 16 comments · Fixed by #8239
Assignees
Labels
bug Unintended code behaviour missing reproduction P1 MUST be fixed or reviewed watchtower

Comments

@pcwd
Copy link

pcwd commented Apr 8, 2022

Background

Remote watchtower has Sequence number error and no longer accepting updates.

2022-04-08 11:01:58.104 [WRN] WTCL: (legacy) SessionQueue(02b72d81698744befe8326d24068213a5c01e57d3c0aa624e40b7b3eb096e4f298) unable to upload state update to tower=02d651a0e8ca200c2b6933029044ff75dafe44363cfc58b8c41de0bbcd7b1150f7@212.86.32.106:9911: received error code StateUpdateCodeSeqNumOutOfOrder in StateUpdateReply for seqnum=1021
2022-04-08 11:01:58.104 [ERR] WTCL: (legacy) SessionQueue(02b72d81698744befe8326d24068213a5c01e57d3c0aa624e40b7b3eb096e4f298) unable to send state update: received error code StateUpdateCodeSeqNumOutOfOrder in StateUpdateReply for seqnum=1021

Trying to remove the misbehaving watchtower gives the following error:

lightning@Lightning:~$ lncli wtclient remove 02d651a0e8ca200c2b6933029044ff75dafe44363cfc58b8c41de0bbcd7b1150f7
[lncli] rpc error: code = Unknown desc = tower has unacked updates

Your environment

  • version of lnd

  • LND 0.14.2-beta

  • which operating system (uname -a on *Nix)

  • Ubuntu 20.04LTS

  • version of btcd, bitcoind, or other backend

  • Bitcoin Core 22.0

Watchtower no longer accepting channel state updates. Can't remove the misbehaving watchtower. LND only uses the first watchtower in the list, so the other watchtowers aren't being used, so, not being able to remove the misbehaving watchtower, leaves me with no watchtower support.

List of watchtowers I have.
{
"towers": [
{
"pubkey": "02d651a0e8ca200c2b6933029044ff75dafe44363cfc58b8c41de0bbcd7b1150f7",
"addresses": [
"212.86.32.106:9911"
],
"active_session_candidate": true,
"num_sessions": 45,
"sessions": [
]
},
{
"pubkey": "03ab82d182a7a2f2a0ecbb0c73ff35adf2211b3adc5cf83cd5b5781cd6c168b4a8",
"addresses": [
"173.50.76.53:9911"
],
"active_session_candidate": true,
"num_sessions": 0,
"sessions": [
]
},
{
"pubkey": "0201fb7bc85498408b1b3d915b1a7af18e189bcbdb0b608870555d7ca7c529b153",
"addresses": [
"177.204.239.6:9911"
],
"active_session_candidate": true,
"num_sessions": 0,
"sessions": [
]
}
]
}

@guggero
Copy link
Collaborator

guggero commented Apr 11, 2022

We definitely need more control over towers and a few bug fixes too in that area.
Not sure what's best for you to do now. Maybe try stopping lnd, removing or renaming the wtclient.db and connecting the working towers again? That might require your node to upload all states again, but at least then you'd be covered again.

@Roasbeef Roasbeef added this to the v0.16.0 milestone Apr 11, 2022
@kshartman
Copy link

Removing wtclient.db fixed the problem.

@saubyk saubyk moved this to 📋 Backlog in lnd v0.16.0 Aug 17, 2022
@saubyk saubyk moved this from 📋 Backlog to 🏗 In progress in lnd v0.16.0 Oct 25, 2022
@saubyk saubyk modified the milestones: v0.16.0, v0.16.1 Dec 6, 2022
@saubyk saubyk removed this from lnd v0.16.0 Dec 6, 2022
@saubyk saubyk modified the milestones: v0.16.2, v0.16.1 Feb 14, 2023
@saubyk saubyk modified the milestones: v0.16.1, v0.17.0 Mar 16, 2023
@saubyk saubyk added this to lnd v0.17 Mar 26, 2023
@saubyk saubyk moved this to 📋 Backlog in lnd v0.17 Mar 26, 2023
@saubyk saubyk linked a pull request Apr 20, 2023 that will close this issue
@saubyk saubyk modified the milestones: v0.17.0, v0.17.1 Jun 15, 2023
@saubyk saubyk removed this from lnd v0.17 Jun 15, 2023
@saubyk saubyk added the P1 MUST be fixed or reviewed label Aug 8, 2023
@saubyk saubyk removed this from the High Priority milestone Aug 8, 2023
@saubyk saubyk added this to lnd v0.18 Sep 14, 2023
@saubyk saubyk moved this to 👀 In review in lnd v0.18 Sep 14, 2023
@knox203
Copy link

knox203 commented Oct 15, 2023

Hi, I've been having this issue for awhile myself, I can provide logs if at all helpful.

Main issue for me is not being able to remove a faulty/offline tower, error:
[lncli] rpc error: code = Unknown desc = tower has unacked updates

@ellemouton
Copy link
Collaborator

@knox203 - what version of LND are you running?

@knox203
Copy link

knox203 commented Oct 17, 2023

@knox203 - what version of LND are you running?

Currently running v0.17.0-beta.

@ellemouton
Copy link
Collaborator

hmm ok - could you please provide some extra logs @knox203 ? This should have been fixed with the last release

@knox203
Copy link

knox203 commented Oct 23, 2023

hmm ok - could you please provide some extra logs @knox203 ? This should have been fixed with the last release

Hey @ellemouton, sorry about the delay. One additional thing I should mention is that the problematic tower in question went down a long time ago (used to be the public lightningnetwork.plus tower service).

Here are some logs, I filtered out all the duplicates. It seems to repeat the (legacy) Processing backup tasks over and over again tens of thousands of times.

The logs don't look particularly useful to me, but hopefully something may stick out to you!

lnd[806118]:	 2023-10-22 	15:16:40.343 	[INF] 	LTND: 	Version: 0.17.0-beta commit=v0.17.0-beta, build=production, logging=default, debuglevel=CNCT=info,CRTR=info,HSWC=info,NTFN=info,NANN=info,RPCS=info,WTCL=debug
lnd[806118]:	 2023-10-22 	15:16:40.343 	[INF] 	LTND: 	Active chain: Bitcoin (network=mainnet)
lnd[806118]:	 2023-10-22 	15:16:45.723 	[INF] 	WTCL: 	(legacy) Using private watchtower 023bad37e5795654cecc69b43599da8bd5789ac633c098253f60494bde602b60bf, offering policy (blob-type=10 max-updates=1024 reward-rate=0 sweep-fee-rate=2500)
lnd[806118]:	 2023-10-22 	15:16:45.803 	[INF] 	WTCL: 	(anchor) Using private watchtower 023bad37e5795654cecc69b43599da8bd5789ac633c098253f60494bde602b60bf, offering policy (blob-type=110 max-updates=1024 reward-rate=0 sweep-fee-rate=2500)
lnd[806118]:	 2023-10-22 	15:16:50.839 	[INF] 	WTCL: 	(legacy) Watchtower client starting
lnd[806118]:	 2023-10-22 	15:16:50.840 	[DBG] 	WTCL: 	(legacy) Starting session negotiator
lnd[806118]:	 2023-10-22 	15:16:50.840 	[DBG] 	WTCL: 	(legacy) Starting channel close handler
lnd[806118]:	 2023-10-22 	15:16:50.841 	[INF] 	WTCL: 	(legacy) Watchtower client started successfully
lnd[806118]:	 2023-10-22 	15:16:50.841 	[INF] 	WTCL: 	(anchor) Watchtower client starting
lnd[806118]:	 2023-10-22 	15:16:50.841 	[DBG] 	WTCL: 	(legacy) Starting closable sessions handler
lnd[806118]:	 2023-10-22 	15:16:50.841 	[INF] 	WTCL: 	(anchor) Starting session=0255a76a3d79c4aad3aca49c29a7edb307991be42ff8ffe564b9e1b96f3391aebb to process 1 committed backups
lnd[806118]:	 2023-10-22 	15:16:50.842 	[DBG] 	WTCL: 	(anchor) Starting session negotiator
lnd[806118]:	 2023-10-22 	15:16:50.842 	[INF] 	WTCL: 	(anchor) Watchtower client started successfully
lnd[806118]:	 2023-10-22 	15:16:50.842 	[DBG] 	WTCL: 	(anchor) Starting closable sessions handler
lnd[806118]:	 2023-10-22 	15:16:50.843 	[INF] 	WTCL: 	(anchor) SessionQueue(0255a76a3d79c4aad3aca49c29a7edb307991be42ff8ffe564b9e1b96f3391aebb) attempting to dial tower at iiu4epqzm6cydqhezueenccjlyzrqeruntlzbx47mlmdgfwgtrll66qd.onion:9911
lnd[806118]:	 2023-10-22 	15:16:50.843 	[DBG] 	WTCL: 	(anchor) Starting channel close handler
lnd[806118]:	 2023-10-22 	15:16:51.080	[DBG]	WTCL:	(anchor) Processing backup(b36f9748c1dead59c18d6315cb59604c06c5595ebbd650c3d900e3724bf1f5f2, 23384)
lnd[806118]:	 2023-10-22 	15:16:51.083	[INF]	WTCL:	(anchor) Queued backup(b36f9748c1dead59c18d6315cb59604c06c5595ebbd650c3d900e3724bf1f5f2, 23384) successfully for session 02579fb10ed07e9dc0b0bda7247cb4bac8aa25ac4c58c0399c28cc30769ad5c915
lnd[806118]:	 2023-10-22 	15:16:51.128	[DBG]	WTCL:	(legacy) Processing backup(0579baa1b3fa6ae5aa1dfc9a266eb2f8663ef74f516ec8a4994855627317235a, 2199)
lnd[806118]:	 2023-10-22 	15:16:51.130	[INF]	WTCL:	(legacy) Queued backup(0579baa1b3fa6ae5aa1dfc9a266eb2f8663ef74f516ec8a4994855627317235a, 2199) successfully for session 02e0bb907281407ad874464338a30ec9a72f43b007c0e1ce1d8e530412b48ebeed
lnd[806118]:	 2023-10-22 	15:16:51.236	[DBG]	WTCL:	(anchor) Loaded next candidate session queue id=0255a76a3d79c4aad3aca49c29a7edb307991be42ff8ffe564b9e1b96f3391aebb
lnd[806118]:	 2023-10-22 	15:16:51.239	[DBG]	WTCL:	(anchor) SessionQueue(0255a76a3d79c4aad3aca49c29a7edb307991be42ff8ffe564b9e1b96f3391aebb) deciding to accept backup(b36f9748c1dead59c18d6315cb59604c06c5595ebbd650c3d900e3724bf1f5f2, 23405) seqnum=116 pending=2 max-updates=1024
lnd[806118]:	 2023-10-22 	15:16:51.969	[INF]	WTCL:	not processing task for unregistered channel: 17e021979d39807290c13b9528812a7ef7d5e26299fd92547b2e30f08de1ef09
lnd[806118]:	 2023-10-22 	15:16:51.970	[INF]	WTCL:	not processing task for unregistered channel: 17e021979d39807290c13b9528812a7ef7d5e26299fd92547b2e30f08de1ef09
lnd[806118]:	 2023-10-22 	15:16:51.992	[INF]	WTCL:	(legacy) Ignoring ineligible backup(08b0fe3ea2401b5f5d5725dace1e7552a81a35492fd01ae79d1eeebeb25c20a6, 35866)
lnd[806118]:	 2023-10-22 	15:16:52.002	[INF]	WTCL:	not processing task for unregistered channel: 17e021979d39807290c13b9528812a7ef7d5e26299fd92547b2e30f08de1ef09
lnd[806118]:	 2023-10-22 	15:16:52.820	[DBG]	WTCL:	(legacy) Session 02e0bb907281407ad874464338a30ec9a72f43b007c0e1ce1d8e530412b48ebeed exhausted
lnd[806118]:	 2023-10-22 	15:16:54.437	[DBG]	WTCL:	(anchor) Session 0255a76a3d79c4aad3aca49c29a7edb307991be42ff8ffe564b9e1b96f3391aebb exhausted
lnd[806118]:	 2023-10-22 	15:16:54.437	[INF]	WTCL:	(anchor) Requesting new session.
lnd[806118]:	 2023-10-22 	15:16:54.437	[DBG]	WTCL:	(anchor) Dispatching session negotiation
lnd[806118]:	 2023-10-22 	15:16:54.438	[DBG]	WTCL:	(anchor) Attempting session negotiation with tower=023bad37e5795654cecc69b43599da8bd5789ac633c098253f60494bde602b60bf
lnd[806118]:	 2023-10-22 	15:16:55.912	[DBG]	WTCL:	(legacy) Session 036c435394789ed12474912ca900a88a9f3572890bcd13adf34b06ca417cb2e547 exhausted
lnd[806118]:	 2023-10-22 	15:16:56.228	[DBG]	WTCL:	(legacy) Session 028a95c399b2d4c412650821343021636b84d404d7bb7e142f296ecf529fb02120 exhausted
lnd[806118]:	 2023-10-22 	15:16:59.623	[DBG]	WTCL:	(legacy) Session 03302afb31703272f39580481b4316c4eadaa110ecbb06d38e9766459e17047ef7 exhausted
lnd[806118]:	 2023-10-22 	15:17:03.082	[DBG]	WTCL:	(legacy) Session 03f9e77ea7ac6090ea11fc0c14f64b310e457f6695567aede3cf8a308467bf7953 exhausted
lnd[806118]:	 2023-10-22 	15:17:06.631	[DBG]	WTCL:	(legacy) Session 0376264717fbcc3a581ad284389697d17a9b8729ad8aa254c152b04e8ca2fb0603 exhausted
lnd[806118]:	 2023-10-22 	15:17:06.631	[DBG]	WTCL:	(legacy) Loaded next candidate session queue id=020c661d8f97a5b6cc74ae7852e829c7a5d5fd19e349d72489cb53a6ee9d043dcc
lnd[806118]:	 2023-10-22 	15:17:06.634	[INF]	WTCL:	(legacy) SessionQueue(020c661d8f97a5b6cc74ae7852e829c7a5d5fd19e349d72489cb53a6ee9d043dcc) attempting to dial tower at iiu4epqzm6cydqhezueenccjlyzrqeruntlzbx47mlmdgfwgtrll66qd.onion:9911
lnd[806118]:	 2023-10-22	15:17:07.466	[DBG]	WTCL:	(legacy) Processing backup(9004816df93460b3d8083a951dcff39d37ca424aae090290eb508b016ad7cc86, 105398)
lnd[806118]:	 2023-10-22	15:17:07.467	[DBG]	WTCL:	(legacy) SessionQueue(020c661d8f97a5b6cc74ae7852e829c7a5d5fd19e349d72489cb53a6ee9d043dcc) deciding to accept backup(9004816df93460b3d8083a951dcff39d37ca424aae090290eb508b016ad7cc86, 105398) seqnum=8 pending=206 max-updates=1024
lnd[806118]:	 2023-10-22	15:17:07.468	[INF]	WTCL:	(legacy) Queued backup(9004816df93460b3d8083a951dcff39d37ca424aae090290eb508b016ad7cc86, 105398) successfully for session 020c661d8f97a5b6cc74ae7852e829c7a5d5fd19e349d72489cb53a6ee9d043dcc
lnd[806118]:	 2023-10-22	15:17:07.469	[DBG]	WTCL:	(legacy) Processing backup(b4e8609d5c747cb8c9c5aff0168dd9b59ccaabdae2e30a52cd39ed2614c2f7d4, 16747)
lnd[806118]:	 2023-10-22	15:17:07.469	[INF]	WTCL:	not processing task for unregistered channel: b4e8609d5c747cb8c9c5aff0168dd9b59ccaabdae2e30a52cd39ed2614c2f7d4

@ellemouton
Copy link
Collaborator

@knox203 - are these logs taken from the time where you are calling "Remove Tower"?

@knox203
Copy link

knox203 commented Oct 23, 2023

@knox203 - are these logs taken from the time where you are calling "Remove Tower"?

@ellemouton,

Running: lncli wtclient remove 023bad37e5795654cecc69b43599da8bd5789ac633c098253f60494bde602b60bf

Returns: [lncli] rpc error: code = Unknown desc = tower has unacked updates

And in the log I see:

lnd[806118]: 	2023-10-23 08:42:08.015 	[DBG] 	WTCL: 	(legacy) SessionQueue(03364317769d2ae8139c1bfadda28759be041b11f256e22f5f5ff5fa9017d9a6fd) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.015 	[DBG] 	WTCL: 	(legacy) SessionQueue(03364317769d2ae8139c1bfadda28759be041b11f256e22f5f5ff5fa9017d9a6fd) stopped
lnd[806118]: 	2023-10-23 08:42:08.015 	[DBG] 	WTCL: 	(legacy) SessionQueue(03bd5b7f33e5c81abe3d6fb40709bd4d67e70a056a2641fdde143d12da84e91ca4) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.015 	[DBG] 	WTCL: 	(legacy) SessionQueue(03bd5b7f33e5c81abe3d6fb40709bd4d67e70a056a2641fdde143d12da84e91ca4) stopped
lnd[806118]: 	2023-10-23 08:42:08.015 	[DBG] 	WTCL: 	(legacy) SessionQueue(026999d119aa452828f729e761f33c559c54ba44bc98ef0b850f602b09727ec30a) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.016 	[DBG] 	WTCL: 	(legacy) SessionQueue(026999d119aa452828f729e761f33c559c54ba44bc98ef0b850f602b09727ec30a) stopped
lnd[806118]: 	2023-10-23 08:42:08.016 	[DBG] 	WTCL: 	(legacy) SessionQueue(0286effc3257ceb64a5faa0cfe10f2a021306f18ed299b95401c61a85d853074b6) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.016 	[DBG] 	WTCL: 	(legacy) SessionQueue(0286effc3257ceb64a5faa0cfe10f2a021306f18ed299b95401c61a85d853074b6) stopped
lnd[806118]: 	2023-10-23 08:42:08.016 	[DBG] 	WTCL: 	(legacy) SessionQueue(0300b35ea89d915ee81d35cd5c202c66882f13d1e4f27542cbaf7de54967655ff1) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.017 	[DBG] 	WTCL: 	(legacy) SessionQueue(0300b35ea89d915ee81d35cd5c202c66882f13d1e4f27542cbaf7de54967655ff1) stopped
lnd[806118]: 	2023-10-23 08:42:08.017 	[DBG] 	WTCL: 	(legacy) SessionQueue(03fd001d842400c594a23501d34f9e4391f902e4a7a626cc559aac4ab166366eb8) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.017 	[DBG] 	WTCL: 	(legacy) SessionQueue(03fd001d842400c594a23501d34f9e4391f902e4a7a626cc559aac4ab166366eb8) stopped
lnd[806118]: 	2023-10-23 08:42:08.017 	[DBG] 	WTCL: 	(legacy) SessionQueue(020c661d8f97a5b6cc74ae7852e829c7a5d5fd19e349d72489cb53a6ee9d043dcc) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.018 	[DBG] 	WTCL: 	(legacy) SessionQueue(020c661d8f97a5b6cc74ae7852e829c7a5d5fd19e349d72489cb53a6ee9d043dcc) stopped
lnd[806118]: 	2023-10-23 08:42:08.018 	[DBG] 	WTCL: 	(legacy) SessionQueue(021f2520dc283b34e6d42511cc70a8bddc2058e7872338fcf14d003c658234c0ae) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.018 	[DBG] 	WTCL: 	(legacy) SessionQueue(021f2520dc283b34e6d42511cc70a8bddc2058e7872338fcf14d003c658234c0ae) stopped
lnd[806118]: 	2023-10-23 08:42:08.018 	[DBG] 	WTCL: 	(legacy) SessionQueue(03dd0ce5257075f42b75bec29a7db3ca9481ad1234e78e942921c4aa65fabd921f) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.018 	[DBG] 	WTCL: 	(legacy) SessionQueue(03dd0ce5257075f42b75bec29a7db3ca9481ad1234e78e942921c4aa65fabd921f) stopped
lnd[806118]: 	2023-10-23 08:42:08.018 	[DBG] 	WTCL: 	(legacy) SessionQueue(038d564ab3318e63d577ad580a7b59145c17c6d1da785d64b1419aafae71f2e61c) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.019 	[DBG] 	WTCL: 	(legacy) SessionQueue(038d564ab3318e63d577ad580a7b59145c17c6d1da785d64b1419aafae71f2e61c) stopped
lnd[806118]: 	2023-10-23 08:42:08.019 	[DBG] 	WTCL: 	(legacy) SessionQueue(03d97bb8b93f35c6270dff4022df9eea193106d6ef116f47d3fbf1765904f5825f) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.019 	[DBG] 	WTCL: 	(legacy) SessionQueue(03d97bb8b93f35c6270dff4022df9eea193106d6ef116f47d3fbf1765904f5825f) stopped
lnd[806118]: 	2023-10-23 08:42:08.019 	[DBG] 	WTCL: 	(legacy) SessionQueue(033dd0e38e7ae49122ce80747a33f20c53bdf1b0f6c49ad1dbc849d58cf523cfd4) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.020 	[DBG] 	WTCL: 	(legacy) SessionQueue(033dd0e38e7ae49122ce80747a33f20c53bdf1b0f6c49ad1dbc849d58cf523cfd4) stopped
lnd[806118]: 	2023-10-23 08:42:08.020 	[DBG] 	WTCL: 	(legacy) SessionQueue(035f8a23b984ef96821febb73f3347fbe31b264f68ea098b65ad2eb485a6bfe649) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.020 	[DBG] 	WTCL: 	(legacy) SessionQueue(035f8a23b984ef96821febb73f3347fbe31b264f68ea098b65ad2eb485a6bfe649) stopped
lnd[806118]: 	2023-10-23 08:42:08.020 	[DBG] 	WTCL: 	(legacy) SessionQueue(026b62bde295b8f313317812602a384da223ff161e6f897d6914bcfb6256776497) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.021 	[DBG] 	WTCL: 	(legacy) SessionQueue(026b62bde295b8f313317812602a384da223ff161e6f897d6914bcfb6256776497) stopped
lnd[806118]: 	2023-10-23 08:42:08.021 	[DBG] 	WTCL: 	(legacy) SessionQueue(0376264717fbcc3a581ad284389697d17a9b8729ad8aa254c152b04e8ca2fb0603) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.021 	[DBG] 	WTCL: 	(legacy) SessionQueue(0376264717fbcc3a581ad284389697d17a9b8729ad8aa254c152b04e8ca2fb0603) stopped
lnd[806118]: 	2023-10-23 08:42:08.021 	[DBG] 	WTCL: 	(legacy) SessionQueue(03770ead332b94f87aa30531e7a769345a1520920481b24439b299650c19ce0b78) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.022 	[DBG] 	WTCL: 	(legacy) SessionQueue(03770ead332b94f87aa30531e7a769345a1520920481b24439b299650c19ce0b78) stopped
lnd[806118]: 	2023-10-23 08:42:08.022 	[DBG] 	WTCL: 	(legacy) SessionQueue(0378faa050d37a508500cadfb24828ac969d95d663295a0d0aeff932f1b5eb1963) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.022 	[DBG] 	WTCL: 	(legacy) SessionQueue(0378faa050d37a508500cadfb24828ac969d95d663295a0d0aeff932f1b5eb1963) stopped
lnd[806118]: 	2023-10-23 08:42:08.022 	[DBG] 	WTCL: 	(legacy) SessionQueue(03afc0a5a79272d76f4f9e13a331867f19362696a9bb21cdff3cf0463bde3a6647) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.023 	[DBG] 	WTCL: 	(legacy) SessionQueue(03afc0a5a79272d76f4f9e13a331867f19362696a9bb21cdff3cf0463bde3a6647) stopped
lnd[806118]: 	2023-10-23 08:42:08.023 	[DBG] 	WTCL: 	(legacy) SessionQueue(03f9e77ea7ac6090ea11fc0c14f64b310e457f6695567aede3cf8a308467bf7953) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.024 	[DBG] 	WTCL: 	(legacy) SessionQueue(03f9e77ea7ac6090ea11fc0c14f64b310e457f6695567aede3cf8a308467bf7953) stopped
lnd[806118]: 	2023-10-23 08:42:08.024 	[DBG] 	WTCL: 	(legacy) SessionQueue(03302afb31703272f39580481b4316c4eadaa110ecbb06d38e9766459e17047ef7) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.024 	[DBG] 	WTCL: 	(legacy) SessionQueue(03302afb31703272f39580481b4316c4eadaa110ecbb06d38e9766459e17047ef7) stopped
lnd[806118]: 	2023-10-23 08:42:08.024 	[DBG] 	WTCL: 	(legacy) SessionQueue(0373a4f30b6ae52f6589f620c18df64845a596be1922f1e6f66aa9ef79970671e3) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.024 	[DBG] 	WTCL: 	(legacy) SessionQueue(0373a4f30b6ae52f6589f620c18df64845a596be1922f1e6f66aa9ef79970671e3) stopped
lnd[806118]: 	2023-10-23 08:42:08.025 	[DBG] 	WTCL: 	(legacy) SessionQueue(02d35e1191c05b4bf20f63fb7c8ce43413d8ade24182820de5e88b1a276120cd05) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.025 	[DBG] 	WTCL: 	(legacy) SessionQueue(02d35e1191c05b4bf20f63fb7c8ce43413d8ade24182820de5e88b1a276120cd05) stopped
lnd[806118]: 	2023-10-23 08:42:08.025 	[DBG] 	WTCL: 	(legacy) SessionQueue(02a110217d1d1feca6318b8bc869d5bab62f2e6c5f75557a82bd83cad1e5f9edfa) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.025 	[DBG] 	WTCL: 	(legacy) SessionQueue(02a110217d1d1feca6318b8bc869d5bab62f2e6c5f75557a82bd83cad1e5f9edfa) stopped
lnd[806118]: 	2023-10-23 08:42:08.025 	[DBG] 	WTCL: 	(legacy) SessionQueue(02e0bb907281407ad874464338a30ec9a72f43b007c0e1ce1d8e530412b48ebeed) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.026 	[DBG] 	WTCL: 	(legacy) SessionQueue(02e0bb907281407ad874464338a30ec9a72f43b007c0e1ce1d8e530412b48ebeed) stopped
lnd[806118]: 	2023-10-23 08:42:08.026 	[DBG] 	WTCL: 	(legacy) SessionQueue(039b7f85f3d748766460d0452eb0cc22842332ca60e3da40fcdfb87ab95a8fcbf5) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.026 	[DBG] 	WTCL: 	(legacy) SessionQueue(039b7f85f3d748766460d0452eb0cc22842332ca60e3da40fcdfb87ab95a8fcbf5) stopped
lnd[806118]: 	2023-10-23 08:42:08.026 	[DBG] 	WTCL: 	(legacy) SessionQueue(03475b5b84e1e35ea4859cbf803b764e2b393a8166ccb0b3f57d9c5a8db3f01a61) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.026 	[DBG] 	WTCL: 	(legacy) SessionQueue(03475b5b84e1e35ea4859cbf803b764e2b393a8166ccb0b3f57d9c5a8db3f01a61) stopped
lnd[806118]: 	2023-10-23 08:42:08.026 	[DBG] 	WTCL: 	(legacy) SessionQueue(038d3bfda1212aebf1daa0c7cfebfc4af1eae428fb678012087fee7a2424b81dda) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.027 	[DBG] 	WTCL: 	(legacy) SessionQueue(038d3bfda1212aebf1daa0c7cfebfc4af1eae428fb678012087fee7a2424b81dda) stopped
lnd[806118]: 	2023-10-23 08:42:08.027 	[DBG] 	WTCL: 	(legacy) SessionQueue(03e6cae75da5e39ce825d5f97ec545a69586dd4efc64a689bd6bf7eb36567b58c0) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.027 	[DBG] 	WTCL: 	(legacy) SessionQueue(03e6cae75da5e39ce825d5f97ec545a69586dd4efc64a689bd6bf7eb36567b58c0) stopped
lnd[806118]: 	2023-10-23 08:42:08.027 	[DBG] 	WTCL: 	(legacy) SessionQueue(027e84dd1d87432a2e7aeb5385431f6fa476eaaaa8e43d1025d810b93c7c418325) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.028 	[DBG] 	WTCL: 	(legacy) SessionQueue(027e84dd1d87432a2e7aeb5385431f6fa476eaaaa8e43d1025d810b93c7c418325) stopped
lnd[806118]: 	2023-10-23 08:42:08.028 	[DBG] 	WTCL: 	(legacy) SessionQueue(0398330fdeb0e992d067d565b97e010093898044220040872918c3aadbdd2e8770) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.028 	[DBG] 	WTCL: 	(legacy) SessionQueue(0398330fdeb0e992d067d565b97e010093898044220040872918c3aadbdd2e8770) stopped
lnd[806118]: 	2023-10-23 08:42:08.028 	[DBG] 	WTCL: 	(legacy) SessionQueue(0219fde3e906295a6ebe0e1621ecd93ad83f9d1a7b71fe7b48e39328934942391a) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.029 	[DBG] 	WTCL: 	(legacy) SessionQueue(0219fde3e906295a6ebe0e1621ecd93ad83f9d1a7b71fe7b48e39328934942391a) stopped
lnd[806118]: 	2023-10-23 08:42:08.029 	[DBG] 	WTCL: 	(legacy) SessionQueue(029c4fb3c7ceec39ebef90826dd7fd4c8cb9e7c69f3498e6ddf458c18c8df6c1a9) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.029 	[DBG] 	WTCL: 	(legacy) SessionQueue(029c4fb3c7ceec39ebef90826dd7fd4c8cb9e7c69f3498e6ddf458c18c8df6c1a9) stopped
lnd[806118]: 	2023-10-23 08:42:08.029 	[DBG] 	WTCL: 	(legacy) SessionQueue(038e2046202d1262a3ef3213720f09dc6a8eb568aacfcaceb65eacf8c00d3b6be6) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.030 	[DBG] 	WTCL: 	(legacy) SessionQueue(038e2046202d1262a3ef3213720f09dc6a8eb568aacfcaceb65eacf8c00d3b6be6) stopped
lnd[806118]: 	2023-10-23 08:42:08.030 	[DBG] 	WTCL: 	(legacy) SessionQueue(03ba0bf11076ccaca17a5c6b68f3bcd02d4a19a1e8a9f7e4658fc4722f2edd037e) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.031 	[DBG] 	WTCL: 	(legacy) SessionQueue(03ba0bf11076ccaca17a5c6b68f3bcd02d4a19a1e8a9f7e4658fc4722f2edd037e) stopped
lnd[806118]: 	2023-10-23 08:42:08.031 	[DBG] 	WTCL: 	(legacy) SessionQueue(03cf3d4badc5d44d5083739fab58bc86c491d53c4c42361cc59f97a5e748e26ae9) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.031 	[DBG] 	WTCL: 	(legacy) SessionQueue(03cf3d4badc5d44d5083739fab58bc86c491d53c4c42361cc59f97a5e748e26ae9) stopped
lnd[806118]: 	2023-10-23 08:42:08.031 	[DBG] 	WTCL: 	(legacy) SessionQueue(03f8060f4906a7251a2efa3b8f0d43a1370e45c46d1b349d282ef5d55dc212d483) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.032 	[DBG] 	WTCL: 	(legacy) SessionQueue(03f8060f4906a7251a2efa3b8f0d43a1370e45c46d1b349d282ef5d55dc212d483) stopped
lnd[806118]: 	2023-10-23 08:42:08.032 	[DBG] 	WTCL: 	(legacy) SessionQueue(02d3f75f575c668c3259c71f0577bb62eefe5f64ebd34a00313a4a1a8fab7c8a37) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.032 	[DBG] 	WTCL: 	(legacy) SessionQueue(02d3f75f575c668c3259c71f0577bb62eefe5f64ebd34a00313a4a1a8fab7c8a37) stopped
lnd[806118]: 	2023-10-23 08:42:08.032 	[DBG] 	WTCL: 	(legacy) SessionQueue(02f124b5da261c8e596387e6017a964f995f0aa614ae5c06561b7c4c17c4edd9b5) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.033 	[DBG] 	WTCL: 	(legacy) SessionQueue(02f124b5da261c8e596387e6017a964f995f0aa614ae5c06561b7c4c17c4edd9b5) stopped
lnd[806118]: 	2023-10-23 08:42:08.033 	[DBG] 	WTCL: 	(legacy) SessionQueue(028a95c399b2d4c412650821343021636b84d404d7bb7e142f296ecf529fb02120) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.033 	[DBG] 	WTCL: 	(legacy) SessionQueue(028a95c399b2d4c412650821343021636b84d404d7bb7e142f296ecf529fb02120) stopped
lnd[806118]: 	2023-10-23 08:42:08.033 	[DBG] 	WTCL: 	(legacy) SessionQueue(02b78a2773092d01f4365868c2ca9836dfccb53f9450685d11162a9b7ea9f00ea5) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.033 	[DBG] 	WTCL: 	(legacy) SessionQueue(02b78a2773092d01f4365868c2ca9836dfccb53f9450685d11162a9b7ea9f00ea5) stopped
lnd[806118]: 	2023-10-23 08:42:08.033 	[DBG] 	WTCL: 	(legacy) SessionQueue(036c435394789ed12474912ca900a88a9f3572890bcd13adf34b06ca417cb2e547) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.034 	[DBG] 	WTCL: 	(legacy) SessionQueue(036c435394789ed12474912ca900a88a9f3572890bcd13adf34b06ca417cb2e547) stopped
lnd[806118]: 	2023-10-23 08:42:08.034 	[DBG] 	WTCL: 	(legacy) SessionQueue(03f1676b81b6e7ea191c358a64a1894ae4b7af7cd259d0445f5920ca7046aef8fd) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.034 	[DBG] 	WTCL: 	(legacy) SessionQueue(03f1676b81b6e7ea191c358a64a1894ae4b7af7cd259d0445f5920ca7046aef8fd) stopped
lnd[806118]: 	2023-10-23 08:42:08.034 	[DBG] 	WTCL: 	(legacy) SessionQueue(038c09cbebebccf302022d6503d702b2c83c9506f4c765cea5007d0f01267fb511) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.034 	[DBG] 	WTCL: 	(legacy) SessionQueue(038c09cbebebccf302022d6503d702b2c83c9506f4c765cea5007d0f01267fb511) stopped
lnd[806118]: 	2023-10-23 08:42:08.034 	[DBG] 	WTCL: 	(legacy) SessionQueue(03bfcd093fa7f8ec52dd66cc7e8728684da36f116a78c0f68c0a70e7b425cc92b8) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.034 	[DBG] 	WTCL: 	(legacy) SessionQueue(03bfcd093fa7f8ec52dd66cc7e8728684da36f116a78c0f68c0a70e7b425cc92b8) stopped
lnd[806118]: 	2023-10-23 08:42:08.034 	[DBG] 	WTCL: 	(legacy) SessionQueue(022f9bd572982f07f4e6fb2b207dc78de76aead1b67dad13e3f7e1428a77f8e918) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.035 	[DBG] 	WTCL: 	(legacy) SessionQueue(022f9bd572982f07f4e6fb2b207dc78de76aead1b67dad13e3f7e1428a77f8e918) stopped
lnd[806118]: 	2023-10-23 08:42:08.035 	[DBG] 	WTCL: 	(legacy) SessionQueue(025ac0f86c66984c127c839a0215c92e494c1f4af2af65d98cc17792e736f7c4b4) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.035 	[DBG] 	WTCL: 	(legacy) SessionQueue(025ac0f86c66984c127c839a0215c92e494c1f4af2af65d98cc17792e736f7c4b4) stopped
lnd[806118]: 	2023-10-23 08:42:08.035 	[DBG] 	WTCL: 	(legacy) SessionQueue(026c9a5225b953b3e36d3517016230e80f2adef6f533439acd191f31a688dd293a) stopping ...
lnd[806118]: 	2023-10-23 08:42:08.035 	[DBG] 	WTCL: 	(legacy) SessionQueue(026c9a5225b953b3e36d3517016230e80f2adef6f533439acd191f31a688dd293a) stopped
lnd[806118]: 	2023-10-23 08:42:08.038 	[ERR] 	RPCS: 	[/wtclientrpc.WatchtowerClient/RemoveTower]: tower has unacked updates

@ellemouton
Copy link
Collaborator

@knox203 - do you see any logs (after the remove call) along the following lines?

  • could not re-queue
  • could not delete committed update
  • could not re-queue backup task

Also, does this "tower has unacked updates" error appear consistently? ie, if you retry the call, does it still fail with that?

@ellemouton
Copy link
Collaborator

ahhh ok I think i might know why this could happen. Gonna investigate a bit more & try re-produce & fix 👍

@knox203
Copy link

knox203 commented Nov 26, 2023

@knox203 - do you see any logs (after the remove call) along the following lines?

* `could not re-queue`

* `could not delete committed update`

* `could not re-queue backup task`

Also, does this "tower has unacked updates" error appear consistently? ie, if you retry the call, does it still fail with that?

@ellemouton,
No, unfortunately not seeing any further messagess from WTCL in the logs after getting the "tower has unacked updates" result. The "tower has unacked updates" appears to be consistent.

ahhh ok I think i might know why this could happen. Gonna investigate a bit more & try re-produce & fix 👍

That's great to hear! Please let me know if there's anything else I can provide or do to assist. Thanks!

@ellemouton
Copy link
Collaborator

cool - thanks @knox203 !

Managed to reproduce & put up a fix for an edge case that would lead to this error your seeing: #8233

Hopefully that's the edge case you are seeing 🤞

On the side im also working on a PR that just tackles making tower removal/deactivation more robust and less finicky

@ellemouton
Copy link
Collaborator

@knox203 - the fix for the issue you were seeing has been merged in #8233 & so should fix your problem in the next release :)

@knox203
Copy link

knox203 commented Jan 12, 2024 via email

@github-project-automation github-project-automation bot moved this from 👀 In review to ✅ Done in lnd v0.18 Feb 21, 2024
@ellemouton
Copy link
Collaborator

Hi all,

Once #8239 makes it into a release (likely 0.18.0), then if you experience something like this issue (the OG issue here from the OP), you should be able to manually terminate the problem session using the new lncli session terminate command.

Please comment here if the issue persists (once 0.18.0 has been released and the new command has been tried)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Unintended code behaviour missing reproduction P1 MUST be fixed or reviewed watchtower
Projects
Status: Done
7 participants