Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store ForestCAR.zst files directly in the database #3334

Closed
2 of 3 tasks
lemmih opened this issue Aug 7, 2023 · 0 comments · Fixed by #3439
Closed
2 of 3 tasks

Store ForestCAR.zst files directly in the database #3334

lemmih opened this issue Aug 7, 2023 · 0 comments · Fixed by #3439
Assignees
Labels
Priority: 2 - High Very important and should be addressed ASAP

Comments

@lemmih
Copy link
Contributor

lemmih commented Aug 7, 2023

Issue summary

When importing a CAR, CAR.zst, or ForestCAR.zst file, we should convert it to ForestCAR.zst and place it directly in the database folder. ForestCAR.zst files are smaller and faster than ParityDB. They're read-only and dead information cannot be garbage collected, but that matters little compared to the benefits of using compressed data.

Acceptance criteria:

  • Convert imported CAR and CAR.zst files into ForestCAR.zst. ForestCAR.zst files obviously do not have to be re-converted. Move the file to the database folder.
  • Opening a database should give a block store that is the union of ParityDB and all available ForestCAR.zst files.
  • Add flag --consume-snapshot. It should do the same as --import-snapshot, but it'll move or delete the snapshot.

Other information and links

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: 2 - High Very important and should be addressed ASAP
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants