CHANNELD_AWAITING_LOCKIN and funding_tx is different from what's on blockchain

[telelvis]

I am opening my first channel and it's stuck in CHANNELD_AWAITING_LOCKIN . I did read the FAQ on and tried to reconnect, restart cln etc
Some background when I executed fundchannel I set the feerate=slow and at the bad time, it took ~6 days to confirm and my node was offline few times, but eventually caught up
Also I noticed funding_txid is very different from actual and nowhere to be found, not on blockchain not in listtransactions, I wonder what it means, basically 0e07fbd2136ed077e5310f814ef565d5ca0da4bf26163c1d525c90531046eb00 (reported in cln) != e8c7a9cc15773989f8e3c012d95ba1c578ec2b95c2d10e8b2559fe2ebe24ef3d (actual)

l2c listpeerchannels
{
   "channels": [
      {
         "peer_id": "0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3",
         "peer_connected": true,
         "channel_type": {
            "bits": [
               12
            ],
            "names": [
               "static_remotekey/even"
            ]
         },
         "state": "CHANNELD_AWAITING_LOCKIN",
         "scratch_txid": "aa996ef1db542f94081fa1c65fc916c9fc1574f4270517b2bea863b956985f5a",
         "last_tx_fee_msat": 184000,
         "feerate": {
            "perkw": 255,
            "perkb": 1020
         },
         "owner": "channeld",
         "direction": 0,
         "channel_id": "00eb461053905c521d3c1626bfa40dcad565f54e810f31e577d06e13d2fb070f",
         "funding_txid": "0e07fbd2136ed077e5310f814ef565d5ca0da4bf26163c1d525c90531046eb00",
         "funding_outnum": 1,
         "close_to_addr": "bc1q0eq7dc0paqvd43wpjmmf4ss7363akm2df65a2h",
         "close_to": "00147e41e6e1e1e818dac5c196f69ac21e8ea3db6d4d",
         "private": false,
         "opener": "local",
         "alias": {
            "local": "10639584x11729467x37835"
         },
         "features": [
            "option_static_remotekey"
         ],
         "funding": {
            "local_funds_msat": "1500000000msat",
            "remote_funds_msat": "0msat",
            "pushed_msat": 0
         },
         "to_us_msat": 1500000000,
         "min_to_us_msat": 1500000000,
         "max_to_us_msat": 1500000000,
         "total_msat": 1500000000,
         "fee_base_msat": 1000,
         "fee_proportional_millionths": 10,
         "dust_limit_msat": 546000,
         "max_total_htlc_in_msat": 18446744073709551615,
         "their_reserve_msat": 15000000,
         "our_reserve_msat": 15000000,
         "spendable_msat": 1484456000,
         "receivable_msat": 0,
         "minimum_htlc_in_msat": 0,
         "minimum_htlc_out_msat": 1,
         "maximum_htlc_out_msat": 1485000000,
         "their_to_self_delay": 144,
         "our_to_self_delay": 180,
         "max_accepted_htlcs": 30,
         "state_changes": [],
         "status": [
            "CHANNELD_AWAITING_LOCKIN:Sent reestablish, waiting for theirs"
         ],
         "in_payments_offered": 0,
         "in_offered_msat": 0,
         "in_payments_fulfilled": 0,
         "in_fulfilled_msat": 0,
         "out_payments_offered": 0,
         "out_offered_msat": 0,
         "out_payments_fulfilled": 0,
         "out_fulfilled_msat": 0,
         "htlcs": []
      }
   ]
}

I tried method here https://bitcoin.stackexchange.com/questions/102180/can-a-channel-be-closed-while-the-funding-tx-is-still-stuck-in-the-mempool to locate funding transaction, sql query returns nothing. Perhaps below command should give similar result

l2c listtransactions
{
   "transactions": [
      {
         "hash": "a0c2506137b4790d3a5a707e8b75465286111642567d6fd177a8fae027d975be",
         "rawtx": "010000000001068aba089954e150868b8066a94e1b2b29e03bf9dc95ad53353249eb197cbeba080000000000ffffffff465468a81109ef93532a4c4ba3e543346f6b612d5567958310b943b15562178d0000000000ffffffff4678eb45c0d62294c0846a6262d25cefd02a02b51f4669a3dae600e6f445df8c0000000000ffffffff37a5a2c8a1db7ebba0c8d8404d54e1ab911e32ec6ff1513496eba7511012fe570000000000ffffffff8e3263409b2c5635ffc3e9da6a846b788830ae8ea25712bdfc4835d3603c7b3d0000000000ffffffffafa6eeef7340ba28dd03b5fbda03387a8a53fa83a1a735ce8add30328e09fabf0000000000ffffffff01a459ff030000000017a914c79af36f2ae357574aa1af91e222b680a53bddb4870141eec957717de62038c263488af4d5f0bb05060f92c8e2a915760371503a900085ba816f6f36e2f5b22dce9408e8d0f8a87909cb0ecdfd6061aba118fc2a9f8045010141554f4884c872319d8b66778cb739ae255493c270e25714f539745d5c76427625ded0ca0d13c7ae6038e366e4082e7156f95b36c6f2b6af7df409394984036f4f0101416d671f673d5323df33d78df8c020f06bf1b5bad7961b651f2bac6418469d0360536417b92bbadd3983460afbfcefce123e89a5a5409c95fb7be58576a1900a5a010141f40d82effc404fe7f49acbe30599b119023f21cc5b3d8830d396f9ce121968e4fbd1057249b4107863b5144372b3b8bfb55920669456f910b4dc01a64725a9060101413a86da51cf07308f2e6190bbbd9f4a6f3fc2b099089b74cca4a1cb81b3bef8fb948820b19fdcfc3cddbf04e51de896bf7bee7c67802d89eb503c8e415af18e100101415ce2f1c087753862ca7f4fe33f30893878d3c040b2cf886ce1169d7a488a059f5b97586b630e05fb050792e655c2cb4edfb199fb51bbfa6cce6af85f6c6ea36d0100000000",
         "blockheight": 784777,
         "txindex": 714,
         "locktime": 0,
         "version": 1,
         "inputs": [
            {
               "txid": "08babe7c19eb49323553ad95dcf93be0292b1b4ea966808b8650e1549908ba8a",
               "index": 0,
               "sequence": 4294967295
            },
            {
               "txid": "8d176255b143b910839567552d616b6f3443e5a34b4c2a5393ef0911a8685446",
               "index": 0,
               "sequence": 4294967295
            },
            {
               "txid": "8cdf45f4e600e6daa369461fb5022ad0ef5cd262626a84c09422d6c045eb7846",
               "index": 0,
               "sequence": 4294967295
            },
            {
               "txid": "57fe121051a7eb963451f16fec321e91abe1544d40d8c8a0bb7edba1c8a2a537",
               "index": 0,
               "sequence": 4294967295
            },
            {
               "txid": "3d7b3c60d33548fcbd1257a28eae3088786b846adae9c3ff35562c9b4063328e",
               "index": 0,
               "sequence": 4294967295
            },
            {
               "txid": "bffa098e3230dd8ace35a7a183fa538a7a3803dafbb503dd28ba4073efeea6af",
               "index": 0,
               "sequence": 4294967295
            }
         ],
         "outputs": [
            {
               "index": 0,
               "amount_msat": 67066276000,
               "scriptPubKey": "a914c79af36f2ae357574aa1af91e222b680a53bddb487"
            }
         ]
      },
      {
         "hash": "e8c7a9cc15773989f8e3c012d95ba1c578ec2b95c2d10e8b2559fe2ebe24ef3d",
         "rawtx": "02000000000101be75d927e0faa877d16f7d56421611865246758b7e705a3a0d79b4376150c2a00000000017160014dd677d1be3c2a33f7baa533153fef2865283e6a8fdffffff029275e803000000001600140675c6546564c69a2da2d3f8642b9cef6bf83fb760e31600000000002200201112620802378bdcc1d4e23c4a41b447471bd194042c4e4718272fdbe728e63302473044022032f524937e3e081b5a6ef4976a543384e9d375c635878397196e5dba56d7a61802206b8293fc22f48041f75277a4f228ef65af244c359fcb10ad0a1698bc7039a273012103a6a55e0d225a08a747c185712108371e0e1d0f6ac990f5229b074c69bd39729e8bf90b00",
         "blockheight": 785910,
         "txindex": 1173,
         "locktime": 784779,
         "version": 2,
         "inputs": [
            {
               "txid": "a0c2506137b4790d3a5a707e8b75465286111642567d6fd177a8fae027d975be",
               "index": 0,
               "sequence": 4294967293
            }
         ],
         "outputs": [
            {
               "index": 0,
               "amount_msat": 65566098000,
               "scriptPubKey": "00140675c6546564c69a2da2d3f8642b9cef6bf83fb7"
            },
            {
               "index": 1,
               "amount_msat": 1500000000,
               "scriptPubKey": "00201112620802378bdcc1d4e23c4a41b447471bd194042c4e4718272fdbe728e633"
            }
         ]
      }
   ]
}

this is debug for what happens after ln-cli disconnect true, minus gossipd messages as it's too many

Apr 18 22:17:48 raspberrypi lightningd[1088]: 2023-04-18T21:17:48.562Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 637 perkb to floor of 1274
Apr 18 22:17:51 raspberrypi lightningd[1088]: 2023-04-18T21:17:51.184Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-connectd: discard_peer
Apr 18 22:17:51 raspberrypi lightningd[1088]: 2023-04-18T21:17:51.185Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-lightningd: peer_disconnect_done
Apr 18 22:17:51 raspberrypi lightningd[1088]: 2023-04-18T21:17:51.186Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-lightningd: Will try reconnect in 1 seconds
Apr 18 22:17:51 raspberrypi lightningd[1088]: 2023-04-18T21:17:51.188Z DEBUG   connectd: maybe_free_peer freeing peer!
Apr 18 22:17:51 raspberrypi lightningd[1088]: 2023-04-18T21:17:51.380Z INFO    0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-channeld-chan#1: Peer connection lost
Apr 18 22:17:51 raspberrypi lightningd[1088]: 2023-04-18T21:17:51.380Z INFO    0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-chan#1: Peer transient failure in CHANNELD_AWAITING_LOCKIN: channeld: Owning subdaemon channeld died (62208)
Apr 18 22:17:51 raspberrypi lightningd[1088]: 2023-04-18T21:17:51.519Z DEBUG   plugin-funder: Cleaning up inflights for peer id 0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3
Apr 18 22:17:53 raspberrypi lightningd[1088]: 2023-04-18T21:17:53.120Z DEBUG   connectd: Now try LN connect out for host 3.124.63.44
Apr 18 22:17:53 raspberrypi lightningd[1088]: 2023-04-18T21:17:53.121Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-connectd: Connected out, starting crypto
Apr 18 22:17:53 raspberrypi lightningd[1088]: 2023-04-18T21:17:53.354Z DEBUG   02103f1826be6287c8430e653de06ef16d94b3106c01937a78f5ae2c89bd40f234-hsmd: Got WIRE_HSMD_ECDH_REQ
Apr 18 22:17:53 raspberrypi lightningd[1088]: 2023-04-18T21:17:53.354Z DEBUG   hsmd: Client: Received message 1 from client
Apr 18 22:17:53 raspberrypi lightningd[1088]: 2023-04-18T21:17:53.361Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-connectd: Connect OUT
Apr 18 22:17:53 raspberrypi lightningd[1088]: 2023-04-18T21:17:53.362Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-connectd: peer_out WIRE_INIT
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.250Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-connectd: peer_in WIRE_INIT
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.251Z DEBUG   lightningd: Calling peer_connected hook of plugin chanbackup
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.253Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-connectd: Handed peer, entering loop
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.305Z DEBUG   lightningd: Plugin chanbackup returned from peer_connected hook call
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.305Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-chan#1: Peer has reconnected, state CHANNELD_AWAITING_LOCKIN: connecting subd
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.355Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-channeld-chan#1: pid 9475, msgfd 63
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.355Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-chan#1: Waiting for funding confirmations
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.546Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-chan#1: attempting update blockheight 00eb461053905c521d3c1626bfa40dcad565f54e810f31e577d06e13d2fb070f
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.548Z DEBUG   hsmd: new_client: 1
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.559Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-hsmd: Got WIRE_HSMD_GET_PER_COMMITMENT_POINT
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.560Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-channeld-chan#1: option_static_remotekey = 1, option_anchor_outputs = 0
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.560Z DEBUG   hsmd: Client: Received message 18 from client
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.562Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-channeld-chan#1: init LOCAL: remote_per_commit = 02ec25f0605df2b655db0c7f95569c10afa61a4113039c672566439269714919fd, old_remote_per_commit = 02ec25f0605df2b655db0c7f95569c10afa61a4113039c672566439269714919fd next_idx_local = 1 next_idx_remote = 1 revocations_received = 0 feerates { SENT_ADD_ACK_REVOCATION:255 } range 319-41468 blockheights { SENT_ADD_ACK_REVOCATION:0 }, our current 786021
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.563Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-hsmd: Got WIRE_HSMD_GET_PER_COMMITMENT_POINT
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.564Z DEBUG   hsmd: Client: Received message 18 from client
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.612Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-channeld-chan#1: peer_out WIRE_CHANNEL_REESTABLISH
Apr 18 22:18:00 raspberrypi lightningd[1088]: 2023-04-18T21:18:00.613Z DEBUG   0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3-channeld-chan#1: billboard: Sent reestablish, waiting for theirs
Apr 18 22:18:21 raspberrypi lightningd[1088]: 2023-04-18T21:18:21.425Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 636 perkb to floor of 1273

getinfo output

l2c getinfo
{
   "id": "02103f1826be6287c8430e653de06ef16d94b3106c01937a78f5ae2c89bd40f234",
   "alias": "node on the boat",
   "color": "006622",
   "num_peers": 1,
   "num_pending_channels": 1,
   "num_active_channels": 0,
   "num_inactive_channels": 0,
   "address": [
      {
         "type": "torv3",
         "address": "ixrprxzlypup3qem3npnel5jtctltewc2bed7r2wnrwcqfv7saynlmad.onion",
         "port": 9735
      }
   ],
   "binding": [
      {
         "type": "ipv4",
         "address": "127.0.0.1",
         "port": 9735
      }
   ],
   "version": "v23.02.2-171-gca80dee",
   "blockheight": 786021,
   "network": "bitcoin",
   "fees_collected_msat": 0,
   "lightning-dir": "/home/telelvis/.lightning/bitcoin",
   "our_features": {
      "init": "08a0000a0269a2",
      "node": "88a0000a0269a2",
      "channel": "",
      "invoice": "02000002024100"
   }
}

Many thanks!

[telelvis]

maybe this will help #4421 , not sure though

[telelvis]

Update on above, as it's on mainnet, I am in touch with the peer node owner and trying to get them cooperate to execute something like: lightning-cli close wrong_funding
Channel still remains in CHANNELD_AWAITING_LOCKIN state, 2016 blocks expire soon, I hope peer can perform backup of channel keys, so at least some recovery procedure remain theoretically possible
This is my understanding of the situation and this is what's going on mainnet.

Now interestingly, I brought back my testnet setup 02cbecba9ab7bb23ae2211f476225c5e884966b159a7b03a854eb5f2d20d94fb2, where I was playing around with the same CLN software version v23.02.2-171-gca80dee, opening channels, having much more flexibility. 1 of 7 channels on this testnet node has been negotiated with, again, funding_tx that is nowhere to be found. At the same time another tx has been broadcasted and confirmed, that I suspect is related, as it's same amount and roughly same time. This channel has been in CHANNELD_AWAITING_LOCKIN for a while and I decided to close it from my side, which resulted in the following messages in the log

2023-04-25T15:53:38.182Z UNUSUAL 031c64a68e6d1b9e50711336d92b434c584ce668b2fae59ee688bd73713fee1569-channeld-chan#2: lnd sent 'internal error': let's give it some space
2023-04-25T15:53:38.185Z INFO    031c64a68e6d1b9e50711336d92b434c584ce668b2fae59ee688bd73713fee1569-chan#2: Peer transient failure in CHANNELD_SHUTTING_DOWN: channeld WARNING: error channel 398b1ad5d2ac6b0e81aa7bd0ee15b42b0513df8212c0e0d0f16ec38d99493008: funding failed due to internal error

Status of this testnet channel on my testnet node 02cbecba9ab7bb23ae2211f476225c5e884966b159a7b03a854eb5f2d20d94fb2c, now remains as below
Scratch_tx which I think suppose to go on the blockchain and close this channel is nowhere to be found too

     {
         "peer_id": "031c64a68e6d1b9e50711336d92b434c584ce668b2fae59ee688bd73713fee1569",
         "peer_connected": true,
         "channel_type": {
            "bits": [
               12
            ],
            "names": [
               "static_remotekey/even"
            ]
         },
         "state": "CHANNELD_SHUTTING_DOWN",
         "scratch_txid": "ac47ad32a601fffd4b8d09cf6ef48a647705c38074b5c594beb63f5e3d40168d",
         "last_tx_fee_msat": 183000,
         "feerate": {
            "perkw": 253,
            "perkb": 1012
         },
         "direction": 0,
         "channel_id": "398b1ad5d2ac6b0e81aa7bd0ee15b42b0513df8212c0e0d0f16ec38d99493008",
         "funding_txid": "093049998dc36ef1d0e0c01282df13052bb415eed07baa810e6bacd2d51a8b39",
         "funding_outnum": 1,
         "close_to_addr": "tb1qufrfyj6m2pwl99r8jp7l8s7wun7a24606ws8vh",
         "close_to": "0014e246924b5b505df29467907df3c3cee4fdd5574f",
         "private": false,
         "opener": "local",
         "closer": "local",
         "alias": {
            "local": "11327949x14197802x22751"
         },
         "features": [
            "option_static_remotekey"
         ],
         "funding": {
            "local_funds_msat": "200000000msat",
            "remote_funds_msat": "0msat",
            "pushed_msat": 0
         },
         "to_us_msat": 200000000,
         "min_to_us_msat": 200000000,
         "max_to_us_msat": 200000000,
         "total_msat": 200000000,
         "fee_base_msat": 1,
         "fee_proportional_millionths": 10,
         "dust_limit_msat": 546000,
         "max_total_htlc_in_msat": 18446744073709551615,
         "their_reserve_msat": 2000000,
         "our_reserve_msat": 2000000,
         "spendable_msat": 197460000,
         "receivable_msat": 0,
         "minimum_htlc_in_msat": 0,
         "minimum_htlc_out_msat": 1,
         "maximum_htlc_out_msat": 198000000,
         "their_to_self_delay": 6,
         "our_to_self_delay": 144,
         "max_accepted_htlcs": 483,
         "state_changes": [
            {
               "timestamp": "2023-04-23T10:02:38.451Z",
               "old_state": "CHANNELD_AWAITING_LOCKIN",
               "new_state": "CHANNELD_SHUTTING_DOWN",
               "cause": "user",
               "message": "User or plugin invoked close command"
            }
         ],
         "status": [
            "CHANNELD_SHUTTING_DOWN:Received error channel 398b1ad5d2ac6b0e81aa7bd0ee15b42b0513df8212c0e0d0f16ec38d99493008: funding failed due to internal error",
            "CHANNELD_SHUTTING_DOWN:Sent reestablish, waiting for theirs"
         ],
         "in_payments_offered": 0,
         "in_offered_msat": 0,
         "in_payments_fulfilled": 0,
         "in_fulfilled_msat": 0,
         "out_payments_offered": 0,
         "out_offered_msat": 0,
         "out_payments_fulfilled": 0,
         "out_fulfilled_msat": 0,
         "htlcs": []
      }

The transaction that I *suspect is related to this channel on testnet is 13a9a88e9f7b3cd65eb99f52821d4816b9b5e55dd204a57995e2e826678f77a8 , output 1

--
My next steps are to wait for peer on mainnet to come back with some new and see if they're willing to cooperate to attempt make up some closure.
And on testnet - continue digging, see if I can find something, install latest stable version, which I should've done in the first place.
If you have any advice, it's more than welcome!