Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store: introduce Database::get_with_rc_stripped method #7483

Merged
merged 3 commits into from
Aug 26, 2022

Conversation

mina86
Copy link
Contributor

@mina86 mina86 commented Aug 25, 2022

Add get_with_rc_stripped method to the Database trait which strips the
reference count from the returned value.

Previously, Store::get method called get_raw_bytes and then (for
reference counted columns) refcount::get_with_rc_logic to strip the
reference count. With this change, it now calls either get_raw_bytes
or get_with_rc_stripped as needed.

The reason for this addition is to help with cold storage. We don’t
want to keep refcounts in cold storage which means that we would need
to append dummy refcount to get_raw_bytes calls against cold storage.
By having a separate method which returns values without refcount, we
won’t need to worry about this dummy count.

As a side benefit, this removes a memory allocation when reading
a reference counted column. Previosly get_payload method was used
to strip the refcount:

fn get_payload(value: Vec<u8>) -> Option<Vec<u8>> {
    decode_value_with_rc(&value).0.map(|v| v.to_vec())
}

The issue with it is that it takes a Vec as argument, takes a slice of
it and creates a new Vec from it. Now the vector is simply truncated
by strip_refcount function.

Add get_with_rc_stripped method to the Database trait which strips the
reference count from the returned value.

Previously, Store::get method called get_raw_bytes and then (for
reference counted columns) refcount::get_with_rc_logic to strip the
reference count.  With this change, it now calls either get_raw_bytes
or get_with_rc_stripped as needed.

The reason for this addition is to help with cold storage.  We don’t
want to keep refcounts in cold storage which means that we would need
to append dummy refcount to get_raw_bytes calls against cold storage.
By having a separate method which returns values without refcount, we
won’t need to worry about this dummy count.

As a side benefit, this removes a memory allocation when reading
a reference counted column.  Previosly get_payload method was used
to strip the refcount:

    fn get_payload(value: Vec<u8>) -> Option<Vec<u8>> {
        decode_value_with_rc(&value).0.map(|v| v.to_vec())
    }

The issue with it is that it takes a Vec as argument, takes a slice of
it and creates a new Vec from it.  Now the vector is simply truncated
by strip_refcount function.
@mina86 mina86 requested a review from a team as a code owner August 25, 2022 22:12
@mina86 mina86 requested a review from matklad August 25, 2022 22:12
core/store/src/db/refcount.rs Outdated Show resolved Hide resolved
@near-bulldozer near-bulldozer bot merged commit 046e1d0 into near:master Aug 26, 2022
@mina86 mina86 deleted the e branch August 26, 2022 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants