Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug]: Size of channel.db and wtclient.db unchanged since last compacting. #8649

Closed
feelancer21 opened this issue Apr 15, 2024 · 11 comments
Closed
Labels
bug Unintended code behaviour needs triage

Comments

@feelancer21
Copy link
Contributor

feelancer21 commented Apr 15, 2024

I compacted the dbs on Saturday. Since then, channel.db and wtclient have not changed. They still have the exact size that was written to the logfile after compacting. Sphinxrelay.db, however, has changed.
I have had about 600,000 state updates since compacting. That's why I don't dare to restart lnd at the moment, because I don't want to end up in a scenario like xmrk with penalties.

Are there things I can check while lnd is running?

Your environment

  • version of lnd
    v0.16.4-beta

  • which operating system (uname -a on *Nix)
    5.10.0-26-amd64 #1 SMP Debian 5.10.197-1 (2023-09-29) x86_64 GNU/Linux

  • version of btcd, bitcoind, or other backend
    bitcoind

@feelancer21 feelancer21 added bug Unintended code behaviour needs triage labels Apr 15, 2024
@guggero
Copy link
Collaborator

guggero commented Apr 15, 2024

I don't think changes to bbolt databases necessarily always directly reflect on disk, since parts of the database are memory mapped directly. So I don't think the size is a very good indicator of a problem.
Do you see any change when you run sync?

@feelancer21
Copy link
Contributor Author

Thanks for quick response.

I run sync channel.db. No idea how to use it correctly. But there was no output.

Usually my db grew about 100MB per day. Now constant for two days with 600k updates and I never seen such a behaviour before.

Wondering if there was a bigger disk space allocation durcing compactation.

@guggero
Copy link
Collaborator

guggero commented Apr 15, 2024

You just run sync to tell the kernel to flush all not-yet-written data to disk. There is no output, that's to be expected. How long did it take approximately? Did it return immediately or take a couple of seconds?

Are you sure it compacted two days ago? After compaction, did you delete a lot of data (e.g. invoices or payments)?

@feelancer21
Copy link
Contributor Author

Returned immediately ...
Sure, it was on two days ago. Found the log entry with the sizes after compactation.

I am running deletepayments daily via batch. There is a four to five digit number of rebalancing trials daily. And I have activated the following options in lnd.conf

gc-canceled-invoices-on-startup=true
gc-canceled-invoices-on-the-fly=true

@feelancer21
Copy link
Contributor Author

And I closed a lot of channels .. Compactation gain was 1.56x

Last compactation was around 4 or 5 weeks before.

@guggero
Copy link
Collaborator

guggero commented Apr 15, 2024

Did you close any channels right after the compaction? Basically what I'm trying to get at: When you delete data in a bbolt database, you don't immediately get that freed up space back in terms of disk usage. That's what compaction is for (it basically copies all data into a new file, omitting any data marked as free space). If you delete something without compacting, it marks the space as free in the bbolt's free list. That space can then be used by new data created and the size of the database is only extended once all free space is used up.
So my question: After the last compaction, did you do any action (manually or explainable by nightly job) that deleted enough data to free up space for 600k updates?

@feelancer21
Copy link
Contributor Author

I closed a lot of channels after the compactation in March. But not after the compactation in April.
And I run deletepayments daily via batch. But not directly after startup.

Is it possible that lnd itself marked something as deleted during the startup process after compactation (e.g. invoices) and now using this free space?

@guggero
Copy link
Collaborator

guggero commented Apr 15, 2024

It's possible yes. But unfortunately there is no way to know without shutting down lnd and inspecting the database with bbolt specific tools.

@feelancer21
Copy link
Contributor Author

I stopped forwarding and rebalancing and compacted a copy of the db with chantools. Channel.db shrinked about another 700MB without any error. Will restart lnd again.

@guggero
Copy link
Collaborator

guggero commented Apr 15, 2024

shrinked about another 700MB without any error

I think that confirms my theory. Something (700 MB of data) was deleted after compaction, allowing the channel states to grow for 700 MB until actually observable through the file size.

With no other indications of something being wrong, I'm closing the issue. Feel free to re-open if anything changes.

@guggero guggero closed this as completed Apr 15, 2024
@feelancer21
Copy link
Contributor Author

Thank you very. I think the reason was that I haven't restarted lnd for one month. I guess it has deleted a big bunch of data. Often I have some restart between the compactations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Unintended code behaviour needs triage
Projects
None yet
Development

No branches or pull requests

2 participants