Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trie: store value of nodes on disk #2323

Closed
qdm12 opened this issue Feb 21, 2022 · 4 comments
Closed

trie: store value of nodes on disk #2323

qdm12 opened this issue Feb 21, 2022 · 4 comments

Comments

@qdm12
Copy link
Contributor

qdm12 commented Feb 21, 2022

Task summary

If #2321 and #2322 do not solve the memory usage issue, this should be worked on.

For now we keep the value of each node of the trie in memory.
We should instead write the nodes values to disk and fetch it from disk when needed.
This will cause a potential speed performance degradation (to verify), but can be a major cause for extreme memory usage.

  • Modify code to store node hash <-> value when creating/modifying a node
  • Modify code to delete node hash <-> value when modifying/removing a node
  • Modify code to fetch node value using the node hash (kept in memory) when needed
  • Memory usage improvement measured and documented
  • Speed performance degradation measured and documented
@qdm12
Copy link
Contributor Author

qdm12 commented Mar 3, 2022

#2340 should remove the need for this, let's hold for now.

@noot
Copy link
Contributor

noot commented Mar 17, 2022

@qdm12 is this still needed? seems like the memory issue is fine now

@qdm12
Copy link
Contributor Author

qdm12 commented Apr 11, 2022

Ehhh, I still need to finish #2310 to know how many nodes we have in memory. In the long run, the node goes out of memory after a few weeks, so this may be needed. Let's keep this opened until we're sure the trie isn't a cause for memory increasing in the long run.

@qdm12
Copy link
Contributor Author

qdm12 commented Dec 7, 2022

This is no longer needed, having the whole state trie (and deltas from forks before finalization) does not use much memory (~100MB at the tip). See #1973 for an update on memory usage.

@qdm12 qdm12 closed this as completed Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants