Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

high memory usage when connected to bitcoind undergoing initial block download #1218

Closed
C-Otto opened this issue May 9, 2018 · 7 comments
Closed
Labels
bitcoind Bitcoin Core backend bug Unintended code behaviour P3 might get fixed, nice to have resource usage Software resource usage improvements/issues

Comments

@C-Otto
Copy link
Contributor

C-Otto commented May 9, 2018

Background

I just installed lnd with bitcoind on mainnet. After bitcoind finished reindexing (due to txindex) , the RAM usage of lnd increased steadily to more than 30 GByte (on a system with 128 GByte RAM).

The last few log messages (trace):

2018-05-09 22:40:50.821 [INF] LNWL: Catching up block hashes to height 521895, this might take a while
2018-05-09 22:40:50.822 [INF] LNWL: Done catching up block hashes
2018-05-09 22:40:50.822 [INF] LNWL: Finished rescan for 0 addresses (synced to block 0000000000000000001d32cb2a1929ff90721e94626c3f44ced62af5b9a530d6, height 521895)
2018-05-09 22:41:54.484 [DBG] LNWL: Possible reorg at block 0000000000000000004134af5968669d654605ecb0836c4424baa45754dfbae7

Your environment

  • version of lnd: lnd version 0.4.1-beta
  • which operating system (uname -a on *Nix): Linux ftp 4.9.0-3-amd64 Fix name typo in README #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
  • version of btcd, bitcoind, or other backend: Bitcoin Core Daemon version v0.16.0

Steps to reproduce

Start bitcoind with reindex and txindex=1, start lnd, wait

Expected behaviour

Less memory consumption. I also notice other log messages now (gossip?) that did not appear before the forced restart.

Actual behaviour

High memory consumption, steadily growing.

@Roasbeef
Copy link
Member

Roasbeef commented May 9, 2018

So you're attempting to sync lnd and bitcoind concurrently? We recommend instead letting bitcoind fully sync, then start up lnd. If you follow this path, then certain optimizations that we put into place to detect the case where bitcoind is already synced will allow your node to boot up much faster.

@C-Otto
Copy link
Contributor Author

C-Otto commented May 9, 2018

I don't know what lnd did, but bitcoind certainly was doing some work (writing lots of data to disk, at least). A restart solved the issue. I still don't think that lnd should consume several GByte of RAM. For this very specific case I'm fine with lack of optimizations, though.

@Roasbeef
Copy link
Member

Roasbeef commented May 9, 2018

I still don't think that lnd should consume several GByte of RAM.

In the normal case it doesn't. I have an lnd node on testnet consuming 20MB as an example.

For this very specific case I'm fine with lack of optimizations, though.

To re-word my comment above: one should allow their bitcoind node to fully sync before starting up lnd.

If you're able to reproduce this, if you run with the --profile flag set, then you can easily extract a memory profile of all the goroutines in the entire daemon. With this, we can then pinpoint why you're seeing high memory usage with lnd connected to a bitcoind that's undergoing the initial block download.

@Roasbeef Roasbeef changed the title Memory Leak high memory usage when connected to bitcoind undergoing initial block download May 9, 2018
@Roasbeef Roasbeef added resource usage Software resource usage improvements/issues bitcoind Bitcoin Core backend bug Unintended code behaviour labels May 9, 2018
@Perlover
Copy link

Perlover commented Jul 2, 2018

I think this bug related with this:

#1299

@Roasbeef Roasbeef added the P3 might get fixed, nice to have label Jul 10, 2018
@C-Otto
Copy link
Contributor Author

C-Otto commented Aug 19, 2018

@Roasbeef
Copy link
Member

Yes the issue is that during IDB, due to the bounded queue of RPC requests, bitcoind can't keep up with out requests. As a result, it drops them and we currently don't retry for those assuming that bitcoind is "trusted". A temp work around would be increasing the size of the RPC work queue.

@wpaulino
Copy link
Contributor

wpaulino commented Oct 2, 2019

No longer an issue as of #3141.

@wpaulino wpaulino closed this as completed Oct 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bitcoind Bitcoin Core backend bug Unintended code behaviour P3 might get fixed, nice to have resource usage Software resource usage improvements/issues
Projects
None yet
Development

No branches or pull requests

4 participants