[bug]: LND consumes a lot of RAM

[starius]

Background

After updating to 1.16.3 I noticed spike in memory usage from about 1G to 2.4G. Number of channels stayed about the same (150).

Also some commands are now slow (lncli listchannels takes sometimes up to a mitute, before it was almost instant). Swap is being used a lot, I think it is related.

Here is pprof memory chart. It has suspicious path eating 90% of all captured memory (cachedInputs.addInput). It mentions mempool. May it be related to recent spike in mempool size?

Your environment

• version of lnd 1.16.3
• which operating system (uname -a on *Nix): Linux vps 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux
• version of bitcoind: 25.0
• any other relevant environment details. The machine has 4G of ram and runs bitcoind and lnd.

Steps to reproduce

I don't know, how. I'm running this node for several years.

Expected behaviour

LND consumes <=1G of RAM.

Actual behaviour

LND consumes 2.4G of RAM.

[Roasbeef]

Fixed by #7767

[begetan]

I have this error after upgrading from v0.15.5 to v0.16.3:

Jun 16 06:20:05 lnd lnd[13479]: 2023-06-16 06:20:05.551 [INF] CRTR: Processed channels=1 updates=53 nodes=4 in last 59.995447853s
Jun 16 06:21:01 lnd lnd[13479]: fatal error: runtime: out of memory
Jun 16 06:21:01 lnd lnd[13479]: runtime stack:
Jun 16 06:21:01 lnd lnd[13479]: runtime.throw({0x168ff56?, 0x0?})
Jun 16 06:21:01 lnd lnd[13479]: #011/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7fd996b6cca0 sp=0x7fd996b6cc70 pc=0x438c3d
Jun 16 06:21:01 lnd lnd[13479]: runtime.sysMapOS(0xc067800000, 0x25400000?)
Jun 16 06:21:01 lnd lnd[13479]: #011/usr/local/go/src/runtime/mem_linux.go:187 +0x11b fp=0x7fd996b6cce8 sp=0x7fd996b6cca0 pc=0x419e3b
Jun 16 06:21:01 lnd lnd[13479]: runtime.sysMap(0x2430320?, 0xc3ffffffff?, 0x24404b8?)
Jun 16 06:21:01 lnd lnd[13479]: #011/usr/local/go/src/runtime/mem.go:142 +0x35 fp=0x7fd996b6cd18 sp=0x7fd996b6cce8 pc=0x419815
Jun 16 06:21:01 lnd lnd[13479]: runtime.(*mheap).grow(0x2430320, 0x12980?)
Jun 16 06:21:01 lnd lnd[13479]: #011/usr/local/go/src/runtime/mheap.go:1522 +0x252 fp=0x7fd996b6cd90 sp=0x7fd996b6cd18 pc=0x429fb2
Jun 16 06:21:01 lnd lnd[13479]: runtime.(*mheap).allocSpan(0x2430320, 0x12980, 0x0, 0x92?)
Jun 16 06:21:01 lnd lnd[13479]: #011/usr/local/go/src/runtime/mheap.go:1243 +0x1b7 fp=0x7fd996b6ce28 sp=0x7fd996b6cd90 pc=0x4296f7
Jun 16 06:21:01 lnd lnd[13479]: runtime.(*mheap).alloc.func1()
Jun 16 06:21:01 lnd lnd[13479]: #011/usr/local/go/src/runtime/mheap.go:961 +0x65 fp=0x7fd996b6ce70 sp=0x7fd996b6ce28 pc=0x4291a5
Jun 16 06:21:01 lnd lnd[13479]: runtime.systemstack()...
the stack trace is enormous, with more than 1000 lines

Usually, my lnd utilizes 500MB of 2GB
Version v0.15.5 works like a charm!

@Roasbeef I wonder if the latest fix addresses this issue too? Basically, was it introduced in 0.16.x ?

[Roasbeef]

Closing as the regression is fixed in #7767 which was merged.

@begetan in later versions, in order to protect from certain mempool related attacks (attacker makes things difficult to confirm) we needed to start to hold extra state to be able to watch the mempool for spends. The upcoming v0.16.4 release optimizes this further. The trade off here is extra memory for increased security (ensure HTLCs can timeout, etc).

Are you able to add swap to your machine? Also which Go version are you using?

For v0.16.x, you can also increase this value as well to reduce CPU usage a bit: bitcoind.txpollinginterval=5m.

In future versions, if a user is running with a newer version of bitcoind, then we can use a new RPC call to make things more memory efficient again.

[begetan]

@Roasbeef
I am using go 1.20.5 now. I've added swap file, despite of it was never needed before. I will monitor the new version with the fix. Hope you will add more optimizations soon.

CPU usage seems to be usual.

[starius]

The fix helped me. After about one day of running LND uses ~700M of RAM, which is great!

Thank you!