Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate data locality of chainbase allocator #974

Open
Tracked by #886
greg7mdp opened this issue Oct 24, 2024 · 0 comments
Open
Tracked by #886

Investigate data locality of chainbase allocator #974

greg7mdp opened this issue Oct 24, 2024 · 0 comments
Assignees

Comments

@greg7mdp
Copy link
Contributor

greg7mdp commented Oct 24, 2024

As the memory usage of the chainbase db is growing rapidly, we would like for Spring to be able to work well even when the available RAM is less that the amount of memory allocated in the chainbase db.

In order to have a better understanding of the current memory usage patterns in Spring, we propose to implement a graphical visualization of the memory as a square map of pages, whith a color coding indicating the percentage of the page that is allocated (bright red would be 100% allocated, bright green would be 0% allocated).

If the display window is 1024 x 1024 pixels, and we want to be able to display the allocation pattern in 512 gb of memory, each pixel would represent 512 kb of memory, or 128 4k pages. The visualization will provide a zoom functionality, so the allocation of individual pages can be visualized.

Also, we plan to investigate whether it would be possible to visualize the access pattern into the memory from Spring as it is running. It may provide insights about possible reorganization of the storage patterns, so that frequently accessed memory has better locality.

@greg7mdp greg7mdp self-assigned this Oct 24, 2024
@enf-ci-bot enf-ci-bot moved this to Todo in Team Backlog Oct 24, 2024
@greg7mdp greg7mdp added 👍 lgtm and removed triage labels Oct 24, 2024
@greg7mdp greg7mdp added this to the Spring v1.0.3 milestone Oct 24, 2024
@greg7mdp greg7mdp moved this from Todo to In Progress in Team Backlog Oct 24, 2024
@BenjaminGormanPMP BenjaminGormanPMP moved this from In Progress to Awaiting Review in Team Backlog Dec 3, 2024
@BenjaminGormanPMP BenjaminGormanPMP moved this from Awaiting Review to Done in Team Backlog Dec 3, 2024
@bhazzard bhazzard removed this from the Spring v1.1.0-rc1 milestone Jan 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants