From 8848dea617a7f88980633ee4549a27f234d684ba Mon Sep 17 00:00:00 2001 From: t-bast Date: Fri, 16 Feb 2024 11:24:56 +0100 Subject: [PATCH 1/2] Activate dual funding by default And update the release notes. --- docs/release-notes/eclair-vnext.md | 55 ++++++++++++++++++- eclair-core/src/main/resources/reference.conf | 2 +- .../blockchain/DummyOnChainWallet.scala | 2 + .../eclair/integration/IntegrationSpec.scala | 2 + .../basic/TwoNodesIntegrationSpec.scala | 4 +- .../basic/fixtures/MinimalNodeFixture.scala | 55 +++++++++---------- .../basic/payment/OfferPaymentSpec.scala | 7 +-- 7 files changed, 90 insertions(+), 37 deletions(-) diff --git a/docs/release-notes/eclair-vnext.md b/docs/release-notes/eclair-vnext.md index 66e1c0a24d..2cdbf9f4c1 100644 --- a/docs/release-notes/eclair-vnext.md +++ b/docs/release-notes/eclair-vnext.md @@ -4,6 +4,21 @@ ## Major changes +### Dual funding + +After many years of work and refining the protocol, [dual funding](https://github.com/lightning/bolts/pull/851) has been added to the BOLTs. + +This release of eclair activates this new feature, that will be used with [cln](https://github.com/ElementsProject/lightning/) nodes automatically. When opening channels to nodes that don't support dual funding, the older funding protocol will be used automatically. + +One of the immediate benefits of dual funding is that the funding transaction can now be RBF-ed, using the `rbfopen` RPC. + +There is currently no way to automatically add funds to channels that are being opened to your node, as deciding whether to do so or not really depends on each node operator's peering strategy. We are however working on plugin examples that node operators can fork to implement their own strategy for contributing to inbound channels. + +### Update minimal version of Bitcoin Core + +With this release, eclair requires using Bitcoin Core 24.1. +Newer versions of Bitcoin Core may be used, but haven't been extensively tested. + ### Use priority instead of block target for feerates Eclair now uses a `slow`/`medium`/`fast` notation for feerates (in the style of mempool.space), @@ -75,7 +90,45 @@ This feature leaks a bit of information about the balance when the channel is al ### Miscellaneous improvements and bug fixes - +#### Use bitcoinheaders.net v2 + +Eclair uses as one of its sources of blockchain data to detect when our node is being eclipsed. +The format of this service is changing, and the older format will be deprecated soon. +We thus encourage eclair nodes to update to ensure that they still have access to this blockchain watchdog. + +#### Force-closing anchor channels fee management + +Various improvements have been made to force-closing channels that need fees to be attached using CPFP or RBF. +Those changes ensure that eclair nodes don't end up paying unnecessarily high fees to force-close channels, even when the mempool is full. + +#### Improve DB usage when closing channels + +When channels that have relayed a lot of HTLCs are closed, we can forget the revocation data for all of those HTLCs and free up space in our DB. We previously did that synchronously, which meant deleting potentially millions of rows synchronously. This isn't a high priority task, so we're now asynchronously deleting that data in smaller batches. + +Node operators can control the rate at which that data is deleted by updating the following values in `eclair.conf`: + +```conf +// During normal channel operation, we need to store information about past HTLCs to be able to punish our peer if +// they publish a revoked commitment. Once a channel closes or a splice transaction confirms, we can clean up past +// data (which reduces the size of our DB). Since there may be millions of rows to delete and we don't want to slow +// down the node, we delete those rows in batches at regular intervals. +eclair.db.revoked-htlc-info-cleaner { + // Number of rows to delete per batch: a higher value will clean up the DB faster, but may have a higher impact on performance. + batch-size = 50000 + // Frequency at which batches of rows are deleted: a lower value will clean up the DB faster, but may have a higher impact on performance. + interval = 15 minutes +} +``` + +See for more details. + +#### Correctly unlock wallet inputs during transaction eviction + +When the mempool is full and transactions are evicted, and potentially double-spent, the Bitcoin Core wallet cannot always safely unlock inputs. + +Eclair is now automatically detecting such cases and telling Bitcoin Core to unlock inputs that are safe to use. This ensures that node operators don't end up with unavailable liquidity. + +See and for more details. ## Verifying signatures diff --git a/eclair-core/src/main/resources/reference.conf b/eclair-core/src/main/resources/reference.conf index 5113fe81d4..8f84b034d6 100644 --- a/eclair-core/src/main/resources/reference.conf +++ b/eclair-core/src/main/resources/reference.conf @@ -70,7 +70,7 @@ eclair { option_anchors_zero_fee_htlc_tx = optional option_route_blinding = disabled option_shutdown_anysegwit = optional - option_dual_fund = disabled + option_dual_fund = optional option_quiesce = disabled option_onion_messages = optional option_channel_type = optional diff --git a/eclair-core/src/test/scala/fr/acinq/eclair/blockchain/DummyOnChainWallet.scala b/eclair-core/src/test/scala/fr/acinq/eclair/blockchain/DummyOnChainWallet.scala index 88ed41d36f..6b3e059e27 100644 --- a/eclair-core/src/test/scala/fr/acinq/eclair/blockchain/DummyOnChainWallet.scala +++ b/eclair-core/src/test/scala/fr/acinq/eclair/blockchain/DummyOnChainWallet.scala @@ -137,6 +137,7 @@ class SingleKeyOnChainWallet extends OnChainWallet with OnchainPubkeyCache { val pubkey = privkey.publicKey // We create a new dummy input transaction for every funding request. var inputs = Seq.empty[Transaction] + val published = collection.concurrent.TrieMap.empty[TxId, Transaction] var rolledback = Seq.empty[Transaction] var doubleSpent = Set.empty[TxId] var abandoned = Set.empty[TxId] @@ -187,6 +188,7 @@ class SingleKeyOnChainWallet extends OnChainWallet with OnchainPubkeyCache { override def publishTransaction(tx: Transaction)(implicit ec: ExecutionContext): Future[TxId] = { inputs = inputs :+ tx + published += (tx.txid -> tx) Future.successful(tx.txid) } diff --git a/eclair-core/src/test/scala/fr/acinq/eclair/integration/IntegrationSpec.scala b/eclair-core/src/test/scala/fr/acinq/eclair/integration/IntegrationSpec.scala index 8028532726..7c81b09aab 100644 --- a/eclair-core/src/test/scala/fr/acinq/eclair/integration/IntegrationSpec.scala +++ b/eclair-core/src/test/scala/fr/acinq/eclair/integration/IntegrationSpec.scala @@ -103,6 +103,8 @@ abstract class IntegrationSpec extends TestKitBaseClass with BitcoindService wit s"eclair.features.${ShutdownAnySegwit.rfcName}" -> "optional", s"eclair.features.${ChannelType.rfcName}" -> "optional", s"eclair.features.${RouteBlinding.rfcName}" -> "optional", + // We keep dual-funding disabled in tests, unless explicitly requested, as most of the network doesn't support it yet. + s"eclair.features.${DualFunding.rfcName}" -> "disabled", ).asJava) val withStaticRemoteKey = commonFeatures.withFallback(ConfigFactory.parseMap(Map( diff --git a/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/TwoNodesIntegrationSpec.scala b/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/TwoNodesIntegrationSpec.scala index 3d2a6e6fca..555dc74d58 100644 --- a/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/TwoNodesIntegrationSpec.scala +++ b/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/TwoNodesIntegrationSpec.scala @@ -1,7 +1,7 @@ package fr.acinq.eclair.integration.basic import fr.acinq.bitcoin.scalacompat.{ByteVector32, SatoshiLong} -import fr.acinq.eclair.channel.{DATA_NORMAL, NORMAL, RealScidStatus, WAIT_FOR_FUNDING_CONFIRMED} +import fr.acinq.eclair.channel._ import fr.acinq.eclair.integration.basic.fixtures.MinimalNodeFixture.getPeerChannels import fr.acinq.eclair.integration.basic.fixtures.composite.TwoNodesFixture import fr.acinq.eclair.testutils.FixtureSpec @@ -51,7 +51,7 @@ class TwoNodesIntegrationSpec extends FixtureSpec with IntegrationPatience { val channelId2 = openChannel(bob, alice, 110_000 sat).channelId val channels = getPeerChannels(alice, bob.nodeId) assert(channels.map(_.data.channelId).toSet == Set(channelId1, channelId2)) - channels.foreach(c => assert(c.state == WAIT_FOR_FUNDING_CONFIRMED)) + channels.foreach(c => assert(c.state == WAIT_FOR_DUAL_FUNDING_SIGNED || c.state == WAIT_FOR_DUAL_FUNDING_CONFIRMED)) confirmChannel(alice, bob, channelId1, BlockHeight(420_000), 21) confirmChannel(bob, alice, channelId2, BlockHeight(420_000), 22) getPeerChannels(bob, alice.nodeId).foreach(c => assert(c.data.isInstanceOf[DATA_NORMAL])) diff --git a/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/fixtures/MinimalNodeFixture.scala b/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/fixtures/MinimalNodeFixture.scala index df0c6961ad..8af4bf3608 100644 --- a/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/fixtures/MinimalNodeFixture.scala +++ b/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/fixtures/MinimalNodeFixture.scala @@ -10,7 +10,7 @@ import com.typesafe.config.ConfigFactory import fr.acinq.bitcoin.scalacompat.Crypto.PublicKey import fr.acinq.bitcoin.scalacompat.{Block, ByteVector32, Satoshi, SatoshiLong, Transaction, TxId} import fr.acinq.eclair.ShortChannelId.txIndex -import fr.acinq.eclair.blockchain.DummyOnChainWallet +import fr.acinq.eclair.blockchain.SingleKeyOnChainWallet import fr.acinq.eclair.blockchain.bitcoind.ZmqWatcher import fr.acinq.eclair.blockchain.bitcoind.ZmqWatcher.{WatchFundingConfirmed, WatchFundingConfirmedTriggered, WatchFundingDeeplyBuried, WatchFundingDeeplyBuriedTriggered} import fr.acinq.eclair.blockchain.fee.{FeeratePerKw, FeeratesPerKw} @@ -30,7 +30,7 @@ import fr.acinq.eclair.payment.relay.{ChannelRelayer, PostRestartHtlcCleaner, Re import fr.acinq.eclair.payment.send.PaymentInitiator import fr.acinq.eclair.router.Router import fr.acinq.eclair.wire.protocol.IPAddress -import fr.acinq.eclair.{BlockHeight, MilliSatoshi, NodeParams, RealShortChannelId, SubscriptionsComplete, TestBitcoinCoreClient, TestDatabases} +import fr.acinq.eclair.{BlockHeight, MilliSatoshi, NodeParams, SubscriptionsComplete, TestBitcoinCoreClient, TestDatabases} import org.scalatest.concurrent.{Eventually, IntegrationPatience} import org.scalatest.{Assertions, EitherValues} @@ -56,7 +56,7 @@ case class MinimalNodeFixture private(nodeParams: NodeParams, offerManager: typed.ActorRef[OfferManager.Command], postman: typed.ActorRef[Postman.Command], watcher: TestProbe, - wallet: DummyOnChainWallet, + wallet: SingleKeyOnChainWallet, bitcoinClient: TestBitcoinCoreClient) { val nodeId = nodeParams.nodeId val routeParams = nodeParams.routerConf.pathFindingExperimentConf.experiments.values.head.getDefaultRouteParams @@ -88,7 +88,7 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat val readyListener = TestProbe("ready-listener") system.eventStream.subscribe(readyListener.ref, classOf[SubscriptionsComplete]) val bitcoinClient = new TestBitcoinCoreClient() - val wallet = new DummyOnChainWallet() + val wallet = new SingleKeyOnChainWallet() val watcher = TestProbe("watcher") val triggerer = TestProbe("payment-triggerer") val watcherTyped = watcher.ref.toTyped[ZmqWatcher.Command] @@ -187,14 +187,15 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat } def fundingTx(node: MinimalNodeFixture, channelId: ByteVector32)(implicit system: ActorSystem): Transaction = { - val fundingTxid = getChannelData(node, channelId).asInstanceOf[ChannelDataWithCommitments].commitments.latest.fundingTxId - node.wallet.funded(fundingTxid) + getChannelData(node, channelId).asInstanceOf[ChannelDataWithCommitments].commitments.latest.localFundingStatus.signedTx_opt.get } def confirmChannel(node1: MinimalNodeFixture, node2: MinimalNodeFixture, channelId: ByteVector32, blockHeight: BlockHeight, txIndex: Int)(implicit system: ActorSystem): Option[RealScidStatus.Temporary] = { - assert(getChannelState(node1, channelId) == WAIT_FOR_FUNDING_CONFIRMED) - val data1Before = getChannelData(node1, channelId).asInstanceOf[DATA_WAIT_FOR_FUNDING_CONFIRMED] - val fundingTx = data1Before.fundingTx_opt.get + val fundingTx = getChannelData(node1, channelId) match { + case d: DATA_WAIT_FOR_DUAL_FUNDING_SIGNED => d.signingSession.fundingTx.tx.buildUnsignedTx() + case d: DATA_WAIT_FOR_DUAL_FUNDING_CONFIRMED => d.latestFundingTx.sharedTx.tx.buildUnsignedTx() + case d => fail(s"unexpected channel=$d") + } val watch1 = node1.watcher.fishForMessage() { case w: WatchFundingConfirmed if w.txId == fundingTx.txid => true; case _ => false }.asInstanceOf[WatchFundingConfirmed] val watch2 = node2.watcher.fishForMessage() { case w: WatchFundingConfirmed if w.txId == fundingTx.txid => true; case _ => false }.asInstanceOf[WatchFundingConfirmed] @@ -218,8 +219,7 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat def confirmChannelDeep(node1: MinimalNodeFixture, node2: MinimalNodeFixture, channelId: ByteVector32, blockHeight: BlockHeight, txIndex: Int)(implicit system: ActorSystem): RealScidStatus.Final = { assert(getChannelState(node1, channelId) == NORMAL) val data1Before = getChannelData(node1, channelId).asInstanceOf[DATA_NORMAL] - val fundingTxid = data1Before.commitments.latest.fundingTxId - val fundingTx = node1.wallet.funded(fundingTxid) + val fundingTx = data1Before.commitments.latest.localFundingStatus.signedTx_opt.get val watch1 = node1.watcher.fishForMessage() { case w: WatchFundingDeeplyBuried if w.txId == fundingTx.txid => true; case _ => false }.asInstanceOf[WatchFundingDeeplyBuried] val watch2 = node2.watcher.fishForMessage() { case w: WatchFundingDeeplyBuried if w.txId == fundingTx.txid => true; case _ => false }.asInstanceOf[WatchFundingDeeplyBuried] @@ -269,17 +269,6 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat sender.expectMsgType[Router.Data] } - /** - * Computes a deterministic [[RealShortChannelId]] based on a txid. We need this so that watchers can verify - * transactions in a independent and stateless fashion, since there is no actual blockchain in those tests. - */ - def deterministicShortId(txId: TxId): RealShortChannelId = { - val blockHeight = txId.value.take(3).toInt(signed = false) - val txIndex = txId.value.takeRight(2).toInt(signed = false) - val outputIndex = 0 // funding txs created by the dummy wallet used in tests only have one output - RealShortChannelId(BlockHeight(blockHeight), txIndex, outputIndex) - } - /** All known funding txs (we don't evaluate immediately because new ones could be created) */ def knownFundingTxs(nodes: MinimalNodeFixture*): () => Iterable[Transaction] = () => nodes.flatMap(_.wallet.published.values) @@ -306,9 +295,9 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat Behaviors.withTimers { timers => Behaviors.receiveMessagePartial { case vr: ZmqWatcher.ValidateRequest => - val res = knownFundingTxs().find(tx => deterministicShortId(tx.txid) == vr.ann.shortChannelId) match { + val res = knownFundingTxs().find(tx => deterministicTxCoordinates(tx.txid) == (vr.ann.shortChannelId.blockHeight, txIndex(vr.ann.shortChannelId))) match { case Some(fundingTx) => Right(fundingTx, ZmqWatcher.UtxoStatus.Unspent) - case None => Left(new RuntimeException(s"unknown realScid=${vr.ann.shortChannelId}, known=${knownFundingTxs().map(tx => deterministicShortId(tx.txid)).mkString(",")}")) + case None => Left(new RuntimeException(s"unknown realScid=${vr.ann.shortChannelId}, known=${knownFundingTxs().map(tx => deterministicTxCoordinates(tx.txid)).mkString(",")}")) } vr.replyTo ! ZmqWatcher.ValidateResult(vr.ann, res) Behaviors.same @@ -319,16 +308,16 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat } Behaviors.same case watch: ZmqWatcher.WatchFundingConfirmed if confirm => - val realScid = deterministicShortId(watch.txId) + val (blockHeight, txIndex) = deterministicTxCoordinates(watch.txId) knownFundingTxs().find(_.txid == watch.txId) match { - case Some(fundingTx) => watch.replyTo ! ZmqWatcher.WatchFundingConfirmedTriggered(realScid.blockHeight, txIndex(realScid), fundingTx) + case Some(fundingTx) => watch.replyTo ! ZmqWatcher.WatchFundingConfirmedTriggered(blockHeight, txIndex, fundingTx) case None => timers.startSingleTimer(watch, 10 millis) } Behaviors.same case watch: ZmqWatcher.WatchFundingDeeplyBuried if deepConfirm => - val realScid = deterministicShortId(watch.txId) + val (blockHeight, txIndex) = deterministicTxCoordinates(watch.txId) knownFundingTxs().find(_.txid == watch.txId) match { - case Some(fundingTx) => watch.replyTo ! ZmqWatcher.WatchFundingDeeplyBuriedTriggered(realScid.blockHeight, txIndex(realScid), fundingTx) + case Some(fundingTx) => watch.replyTo ! ZmqWatcher.WatchFundingDeeplyBuriedTriggered(blockHeight, txIndex, fundingTx) case None => timers.startSingleTimer(watch, 10 millis) } Behaviors.same @@ -338,6 +327,16 @@ object MinimalNodeFixture extends Assertions with Eventually with IntegrationPat } } } + + /** + * We don't use a blockchain in this test setup, but we want to simulate channels being confirmed. + * We choose a block height and transaction index at which the channel confirms deterministically from its txid. + */ + def deterministicTxCoordinates(txId: TxId): (BlockHeight, Int) = { + val blockHeight = txId.value.take(3).toInt(signed = false) + val txIndex = txId.value.takeRight(2).toInt(signed = false) + (BlockHeight(blockHeight), txIndex) + } } def sendPayment(node1: MinimalNodeFixture, amount: MilliSatoshi, invoice: Bolt11Invoice)(implicit system: ActorSystem): Either[PaymentFailed, PaymentSent] = { diff --git a/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/payment/OfferPaymentSpec.scala b/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/payment/OfferPaymentSpec.scala index e45c3c0c75..4a8de21eb5 100644 --- a/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/payment/OfferPaymentSpec.scala +++ b/eclair-core/src/test/scala/fr/acinq/eclair/integration/basic/payment/OfferPaymentSpec.scala @@ -34,14 +34,11 @@ import fr.acinq.eclair.payment._ import fr.acinq.eclair.payment.offer.OfferManager import fr.acinq.eclair.payment.receive.MultiPartHandler.{DummyBlindedHop, ReceivingRoute} import fr.acinq.eclair.payment.send.PaymentInitiator.{SendPaymentToNode, SendSpontaneousPayment} -import fr.acinq.eclair.payment.send.{ClearRecipient, OfferPayment, PaymentLifecycle} -import fr.acinq.eclair.router.Router +import fr.acinq.eclair.payment.send.{OfferPayment, PaymentLifecycle} import fr.acinq.eclair.testutils.FixtureSpec import fr.acinq.eclair.wire.protocol.OfferTypes.{Offer, OfferPaths} -import fr.acinq.eclair.wire.protocol.{IncorrectOrUnknownPaymentDetails, InvalidOnionBlinding} -import fr.acinq.eclair.{CltvExpiry, CltvExpiryDelta, Features, MilliSatoshi, MilliSatoshiLong, ShortChannelId, randomBytes32, randomKey} import fr.acinq.eclair.wire.protocol.{IncorrectOrUnknownPaymentDetails, InvalidOnionBlinding, OfferTypes} -import fr.acinq.eclair.{CltvExpiryDelta, Features, MilliSatoshi, MilliSatoshiLong, randomBytes32, randomKey} +import fr.acinq.eclair.{CltvExpiryDelta, Features, MilliSatoshi, MilliSatoshiLong, ShortChannelId, randomBytes32, randomKey} import org.scalatest.concurrent.IntegrationPatience import org.scalatest.{Tag, TestData} import scodec.bits.HexStringSyntax From e551ecf7846470a040c1011f8b9b4762591c7c30 Mon Sep 17 00:00:00 2001 From: t-bast Date: Wed, 28 Feb 2024 14:27:28 +0100 Subject: [PATCH 2/2] Dual funding release notes - clarify wording around activation - link to sample plugin --- docs/release-notes/eclair-vnext.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/release-notes/eclair-vnext.md b/docs/release-notes/eclair-vnext.md index 2cdbf9f4c1..17e4cc1c7b 100644 --- a/docs/release-notes/eclair-vnext.md +++ b/docs/release-notes/eclair-vnext.md @@ -8,11 +8,12 @@ After many years of work and refining the protocol, [dual funding](https://github.com/lightning/bolts/pull/851) has been added to the BOLTs. -This release of eclair activates this new feature, that will be used with [cln](https://github.com/ElementsProject/lightning/) nodes automatically. When opening channels to nodes that don't support dual funding, the older funding protocol will be used automatically. +This release of eclair activates dual funding, and it will automatically be used with [cln](https://github.com/ElementsProject/lightning/) nodes. When opening channels to nodes that don't support dual funding, the older funding protocol will be used automatically. One of the immediate benefits of dual funding is that the funding transaction can now be RBF-ed, using the `rbfopen` RPC. -There is currently no way to automatically add funds to channels that are being opened to your node, as deciding whether to do so or not really depends on each node operator's peering strategy. We are however working on plugin examples that node operators can fork to implement their own strategy for contributing to inbound channels. +There is currently no way to automatically add funds to channels that are being opened to your node, as deciding whether to do so or not really depends on each node operator's peering strategy. +We have however created a [sample plugin](https://github.com/ACINQ/eclair-plugins/tree/master/channel-funding) that node operators can fork to implement their own strategy for contributing to inbound channels. ### Update minimal version of Bitcoin Core