multifundchannel - some channels are stuck at CHANNELD_AWAITING_LOCKIN until reconnect

[manreo]

Issue and Steps to Reproduce

Hi,
I've opened a multichannel using this code:

lightning-cli multifundchannel '[
    {
        "id": "03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef",
        "amount": "1500000"
    },
    {
        "id": "03f15fc38f36e88bf1d799994d38e10902318287a53eb413b5e8ab0046d863bace",
        "amount": "1500000"
    },
    {
        "id": "02c197ffa4c2aa4105dd4c4b7279ba1b9061b22910ebbfa759b0001bed9ee48a16",
        "amount": "1500000",
    },
    {
        "id": "032ebdae3314f072acd70e66e18572e6ba27f96918009369585e7cdcf36af3cf9e",
        "amount": "1500000",
    }
    
    
]' 1000perkb

2 of the channels were verified correctly, but the other two were stuck on CHANNELD_AWAITING_LOCKIN. Once I disconnected (e.g. lightning-cli disconnect 02c197ffa4c2aa4105dd4c4b7279ba1b9061b22910ebbfa759b0001bed9ee48a16 true , see below) from one of the problematic peers and then reconnected, the channels instantly became active

Here is a full terminal output for the commands that I ran:

machine@machine-ThinkCentre-M73:~$ lightning-cli listpeers 03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef
{
   "peers": [
      {
         "id": "03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef",
         "connected": true,
         "netaddr": [
            "212.51.156.45:9735"
         ],
         "features": "800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000808a52a1",
         "channels": [
            {
               "state": "CHANNELD_AWAITING_LOCKIN",
               "scratch_txid": "ea2987ea2178a19d414c0bb0f78c3884b105e49c87c71a9602f241361fe6f188",
               "last_tx_fee": "183000msat",
               "last_tx_fee_msat": "183000msat",
               "feerate": {
                  "perkw": 253,
                  "perkb": 1012
               },
               "owner": "channeld",
               "short_channel_id": "752622x1511x0",
               "direction": 0,
               "channel_id": "9e11a56b88caebf453f367cdfafb4435000ba98d79ed638665eff10800ac688a",
               "funding_txid": "8a68ac0008f1ef658663ed798da90b003544fbfacd67f353f4ebca886ba5119e",
               "funding_outnum": 0,
               "close_to_addr": "bc1qmwxpdk4tq9wt0lgnwalwxkcr8rvw8m2g4vvjlq",
               "close_to": "0014db8c16daab015cb7fd13777ee35b0338d8e3ed48",
               "private": false,
               "opener": "local",
               "closer": null,
               "features": [
                  "option_static_remotekey"
               ],
               "funding_allocation_msat": {
                  "03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef": 0,
                  "02ee625bb15e64a4b053c31d111ee608a4300f7661f55d0635a0dd08cab52e9025": 1500000000
               },
               "funding_msat": {
                  "03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef": "0msat",
                  "02ee625bb15e64a4b053c31d111ee608a4300f7661f55d0635a0dd08cab52e9025": "1500000000msat"
               },
               "funding": {
                  "local_msat": "1500000000msat",
                  "remote_msat": "0msat",
                  "pushed_msat": "0msat"
               },
               "msatoshi_to_us": 1500000000,
               "to_us_msat": "1500000000msat",
               "msatoshi_to_us_min": 1500000000,
               "min_to_us_msat": "1500000000msat",
               "msatoshi_to_us_max": 1500000000,
               "max_to_us_msat": "1500000000msat",
               "msatoshi_total": 1500000000,
               "total_msat": "1500000000msat",
               "fee_base_msat": "1000msat",
               "fee_proportional_millionths": 10,
               "dust_limit_satoshis": 546,
               "dust_limit_msat": "546000msat",
               "max_htlc_value_in_flight_msat": 18446744073709551615,
               "max_total_htlc_in_msat": "18446744073709551615msat",
               "their_channel_reserve_satoshis": 15000,
               "their_reserve_msat": "15000000msat",
               "our_channel_reserve_satoshis": 15000,
               "our_reserve_msat": "15000000msat",
               "spendable_msatoshi": 1484460000,
               "spendable_msat": "1484460000msat",
               "receivable_msatoshi": 0,
               "receivable_msat": "0msat",
               "htlc_minimum_msat": 0,
               "minimum_htlc_in_msat": "0msat",
               "minimum_htlc_out_msat": "1msat",
               "maximum_htlc_out_msat": "1485000000msat",
               "their_to_self_delay": 144,
               "our_to_self_delay": 180,
               "max_accepted_htlcs": 30,
               "state_changes": [],
               "status": [
                  "CHANNELD_AWAITING_LOCKIN:Sent reestablish, waiting for theirs"
               ],
               "in_payments_offered": 0,
               "in_msatoshi_offered": 0,
               "in_offered_msat": "0msat",
               "in_payments_fulfilled": 0,
               "in_msatoshi_fulfilled": 0,
               "in_fulfilled_msat": "0msat",
               "out_payments_offered": 0,
               "out_msatoshi_offered": 0,
               "out_offered_msat": "0msat",
               "out_payments_fulfilled": 0,
               "out_msatoshi_fulfilled": 0,
               "out_fulfilled_msat": "0msat",
               "htlcs": []
            }
         ]
      }
   ]
}
machine@machine-ThinkCentre-M73:~$ lightning-cli disconnect 03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef  true
{}
machine@machine-ThinkCentre-M73:~$ lightning-cli connect 03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef
{
   "id": "03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef",
   "features": "800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000808a52a1",
   "direction": "out",
   "address": {
      "type": "ipv4",
      "address": "212.51.156.45",
      "port": 9735
   }
}
machine@machine-ThinkCentre-M73:~$ lightning-cli listpeers 03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef
{
   "peers": [
      {
         "id": "03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef",
         "connected": true,
         "netaddr": [
            "212.51.156.45:9735"
         ],
         "features": "800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000808a52a1",
         "channels": [
            {
               "state": "CHANNELD_NORMAL",
               "scratch_txid": "ea2987ea2178a19d414c0bb0f78c3884b105e49c87c71a9602f241361fe6f188",
               "last_tx_fee": "183000msat",
               "last_tx_fee_msat": "183000msat",
               "feerate": {
                  "perkw": 253,
                  "perkb": 1012
               },
               "owner": "channeld",
               "short_channel_id": "752622x1511x0",
               "direction": 0,
               "channel_id": "9e11a56b88caebf453f367cdfafb4435000ba98d79ed638665eff10800ac688a",
               "funding_txid": "8a68ac0008f1ef658663ed798da90b003544fbfacd67f353f4ebca886ba5119e",
               "funding_outnum": 0,
               "close_to_addr": "bc1qmwxpdk4tq9wt0lgnwalwxkcr8rvw8m2g4vvjlq",
               "close_to": "0014db8c16daab015cb7fd13777ee35b0338d8e3ed48",
               "private": false,
               "opener": "local",
               "closer": null,
               "features": [
                  "option_static_remotekey"
               ],
               "funding_allocation_msat": {
                  "03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef": 0,
                  "02ee625bb15e64a4b053c31d111ee608a4300f7661f55d0635a0dd08cab52e9025": 1500000000
               },
               "funding_msat": {
                  "03cfea6e01d8735d29863d0ecc0f637e5da25c3afb1894e6cb1c25cc9c52dbceef": "0msat",
                  "02ee625bb15e64a4b053c31d111ee608a4300f7661f55d0635a0dd08cab52e9025": "1500000000msat"
               },
               "funding": {
                  "local_msat": "1500000000msat",
                  "remote_msat": "0msat",
                  "pushed_msat": "0msat"
               },
               "msatoshi_to_us": 1500000000,
               "to_us_msat": "1500000000msat",
               "msatoshi_to_us_min": 1500000000,
               "min_to_us_msat": "1500000000msat",
               "msatoshi_to_us_max": 1500000000,
               "max_to_us_msat": "1500000000msat",
               "msatoshi_total": 1500000000,
               "total_msat": "1500000000msat",
               "fee_base_msat": "1000msat",
               "fee_proportional_millionths": 10,
               "dust_limit_satoshis": 546,
               "dust_limit_msat": "546000msat",
               "max_htlc_value_in_flight_msat": 18446744073709551615,
               "max_total_htlc_in_msat": "18446744073709551615msat",
               "their_channel_reserve_satoshis": 15000,
               "their_reserve_msat": "15000000msat",
               "our_channel_reserve_satoshis": 15000,
               "our_reserve_msat": "15000000msat",
               "spendable_msatoshi": 1484460000,
               "spendable_msat": "1484460000msat",
               "receivable_msatoshi": 0,
               "receivable_msat": "0msat",
               "htlc_minimum_msat": 0,
               "minimum_htlc_in_msat": "0msat",
               "minimum_htlc_out_msat": "1msat",
               "maximum_htlc_out_msat": "1485000000msat",
               "their_to_self_delay": 144,
               "our_to_self_delay": 180,
               "max_accepted_htlcs": 30,
               "state_changes": [
                  {
                     "timestamp": "2022-09-04T23:38:03.609Z",
                     "old_state": "CHANNELD_AWAITING_LOCKIN",
                     "new_state": "CHANNELD_NORMAL",
                     "cause": "user",
                     "message": "Lockin complete"
                  }
               ],
               "status": [
                  "CHANNELD_AWAITING_LOCKIN:Reconnected, and reestablished.",
                  "CHANNELD_NORMAL:Funding transaction locked. Channel announced."
               ],
               "in_payments_offered": 0,
               "in_msatoshi_offered": 0,
               "in_offered_msat": "0msat",
               "in_payments_fulfilled": 0,
               "in_msatoshi_fulfilled": 0,
               "in_fulfilled_msat": "0msat",
               "out_payments_offered": 0,
               "out_msatoshi_offered": 0,
               "out_offered_msat": "0msat",
               "out_payments_fulfilled": 0,
               "out_msatoshi_fulfilled": 0,
               "out_fulfilled_msat": "0msat",
               "htlcs": []
            }
         ]
      }
   ]
}

I also have DEBUG log file if needed.
@rustyrussell might be related to this - #2776 ?

Thanks

getinfo output

   "version": "0.11.2",
   "blockheight": 752655,
   "network": "bitcoin",
   "msatoshi_fees_collected": 11702662,
   "fees_collected_msat": "11702662msat",
   "lightning-dir": "/home/machine/.lightning/bitcoin",
   "our_features": {
      "init": "282269a2",
      "node": "800000282269a2",
      "channel": "",
      "invoice": "02000020024100"
   }
}

[niftynei]

maybe #5601

[Coinomatron]

Good thing you posted this issue, had a channel in state 'waiting', instantly confirmed after disconnect command.

Thanks!

[rpallai]

I can confirm this on v23.08-modded: 1 of 2 multifund channels stucked in CHANNELD_AWAITING_LOCKIN: Sent reestablish, waiting for theirs immediately turned into CHANNELD_NORMAL when I reconnected to the peer.