lnd crashes with out of memory #1299
Labels
resource usage
Software resource usage improvements/issues
Comments
|
Do you have swap enabled on your rasp pi? You may need swap in order to be able to boot as the scrypt parameters we use currently for wallet encryption may cause smaller machines to reach to swap. Also looking at the logs: Are you sure this is running 0.4.2? Is the |
|
Generally, I wouldn't really recommend folks run LN nodes on rasp pi's as historically, their hardware hasn't been the most reliable. |
|
I'd never ever run any machine without swap. There's tons of stuff in binaries that will only be used during startup, for example. That memory is much better used as buffers/cache for the OS. |
|
Yes I am sure, I ran 0.4.2. Swap is enabled: While starting the old version 0.4.1 again, I noticed, that the lnd is unresponsive as well with regard to lncli requests. The out of memory exception does though not occurr. After quite some time (several minutes) it starts to behave as expected. BTW better hardware is meanwhile available and the migration of my node is in progress. |
|
Nothing fundamental w.r.t memory usage has changed between 0.4.1 and 0.4.2, if anything, memory usage should have gone down. We have received some reports in the past of our scrypt parmaters not playing nice with rasp pi's, but none of the reports have been reproducible, so not actionable. For example I routinely run a small ish node on testnet that hovers around 30-50MB of memory usage.
Is the |
|
Any updates on this? |
|
Same issue on my RPi3, also 1Gb. Single channel. Started opening second right before the crash. Also before the crash, I've tried to disconnect some peers since I noticed an increasing memory usage. I can only provide now my ssh terminal output when I tried to execute listpeers command. http://paste.debian.net/1028448/ UPD: swap is off; started node again from 320MB ram busy in total (bitcoind + lnd). It crashed when memory usage was oscillating around 600-700 at low and 800-900 at peak. |
|
Very odd, as I have a node that runs with 25-50MB of memory usage no problem, and it also has tens of channels. The initial burst can be high at times though, as not all the peers have switched over to the new query feature which allows them to only send what's necessary. This can be made worse if your node is very slow to process the incoming messages. I would try adding a bit of swap in the short term. |
|
I had same problem (i had many Gb's free memory but first start of Try to increase limits of "max locked memory" by: vi /etc/security/limits.d/99-bitcoin.conf My logs about memory (it's not log file - it's STDERR of lnd process. These logs are not logged in log file): |
|
Here So i see only one reason for out of memory - locked memory |
|
@Roasbeef bitcoind is running locally as the same user as lnd. |
|
Unable to reproduce myself, have had |
|
@Roasbeef , did you try to first time start of lnd after installing? I have same error always in many OSes (CentOS 6, Linux Mint 18.3) in first time after installing. When i run after first crash the lnd it continues work without problem of memory. May be this bug is related with first time graph building of LN.... And when i start first time the lnd i have bitcoind (same user) first time too. The lnd waits of synchronization of bitcoind and in this time i see always same problem (out of memory) |
|
Closing this now as now that the new query sync features have been merged and a few goroutine leaks fixed across the codebase, the initial burst in memory usage upon initial connect is gone (if you connect to "modern" peers), and also the chain notifier block epoch goroutines will no longer leak. |
|
Hello! I occur this again when i installer new node
and my logs: And so on... Here is not full log about creash. If you need please let me know |
What backend were you using? Were you syncing it (if a full node) at the same time you synced |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background
Right after startup, lnd is quite responsive, but after a while lncli commands like
getinfotake longer and longer (up to several minutes). After some more time lnd crashes with an out of memory error. Short before it used more than 1.2GB of ram.Your environment
Steps to reproduce
Just start the LND and wait.
Expected behaviour
The lncli commands are expected to return the requested values immediately. The lnd is expected to not crash.
Actual behaviour
Right after startup, lnd is quite responsive, but after a while lncli commands like
getinfotake longer and longer (up to several minutes). After some more time lnd crashes with an out of memory error. Short before it used more than 1.2GB of ram. See attached logfile lnd-crash.txt.The text was updated successfully, but these errors were encountered: