Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: improve trie prefetching #8205

Merged
merged 18 commits into from
Dec 13, 2022
Merged

Conversation

pugachAG
Copy link
Contributor

@pugachAG pugachAG commented Dec 9, 2022

  • Fix instances of clippy::result_unit_err warning
  • Add explicit assert in tests on timeout while waiting for queue to become empty
  • Avoid single-letter variable names
  • Avoid code duplication in prefetching.0.lock().expect(POISONED_LOCK_ERR) by introducing helper lock function
  • Move reserved_memory to PrefetchSlot
  • Add comments to address some confusion I experienced when reading this code.

Related to #8145 and #7723

@pugachAG pugachAG marked this pull request as ready for review December 12, 2022 10:21
@pugachAG pugachAG requested a review from a team as a code owner December 12, 2022 10:21
@pugachAG pugachAG marked this pull request as draft December 12, 2022 10:29
@pugachAG pugachAG marked this pull request as ready for review December 12, 2022 11:32
Copy link
Collaborator

@akhi3030 akhi3030 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not super familiar with the code but trying to get more into it. So some drive-by comments. Feel free to ignore if they do not make sense.

core/store/src/trie/prefetching_trie_storage.rs Outdated Show resolved Hide resolved
core/store/src/trie/prefetching_trie_storage.rs Outdated Show resolved Hide resolved
runtime/runtime/src/prefetch.rs Outdated Show resolved Hide resolved
runtime/runtime/src/prefetch.rs Outdated Show resolved Hide resolved
runtime/runtime/src/prefetch.rs Outdated Show resolved Hide resolved
runtime/runtime/src/prefetch.rs Outdated Show resolved Hide resolved
runtime/runtime/src/prefetch.rs Outdated Show resolved Hide resolved
runtime/runtime/src/prefetch.rs Outdated Show resolved Hide resolved
pugachAG and others added 2 commits December 12, 2022 13:33
Co-authored-by: Akhilesh Singhania <akhi@near.org>
@pugachAG pugachAG requested a review from akhi3030 December 12, 2022 12:53
Copy link
Contributor

@jakmeier jakmeier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, happy to see the code improving! 😄

I left very few comments, mostly optional ones. Feel free to merge anytime you think it's ready.


No action required but some thoughts:

On a more code-philosophical note, I think PrefetchError is a positive change here but there are two reasons why it was like this.

  • It is a common pattern in Rust code that a method that are designed to fail often under normal circumstances should return ownership of all arguments that it owns. This would allow to retry, for example. But here it's not so important as we can clone all args. And in the current state of the code, we don't make use of the returned resources anyway, so no harm in removing that. But generally speaking, I would argue this is good practice and a patterns that can be useful to know in Rust code.
  • Personally, I only create error enums when there is more than one failure case. I guess it's mostly a matter of taste. But my main argument would be that refactoring from an implicit error to a enum error is trivial when a second variant becomes necessary. But going back can be tricky if the type gets integrated into other types, for example as #[source] of another error. Also, going back to bullet point 1, returning resources is awkward when using a specific error type.

But again, looking at your code changes, it has made things more readable. So in this case I am in favor of making the change. 👍

runtime/runtime/src/lib.rs Show resolved Hide resolved
runtime/runtime/src/lib.rs Show resolved Hide resolved
runtime/runtime/src/lib.rs Show resolved Hide resolved
runtime/runtime/src/lib.rs Show resolved Hide resolved
runtime/runtime/src/prefetch.rs Outdated Show resolved Hide resolved
runtime/runtime/src/prefetch.rs Show resolved Hide resolved
@jakmeier jakmeier removed the request for review from mzhangmzz December 12, 2022 13:50
Copy link
Collaborator

@akhi3030 akhi3030 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. It will be best if someone more familiar with the code also did a review though.

core/store/src/trie/prefetching_trie_storage.rs Outdated Show resolved Hide resolved
runtime/runtime/src/lib.rs Show resolved Hide resolved
self.prefetch_enqueued.inc();
Ok(())
}
fn prefetch_trie_key(&self, trie_key: TrieKey) -> Result<(), PrefetchError> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely not part of this PR but we should improve the API for this function. Currently all callers are calling with small TrieKeys. However, some variants e.g. ContractData can be very large and in such cases, the API should take them by value.

An option might be for this function to accept AccountId instead of TrieKey or for the function to take TrieKey as a reference (but this will introduce a lot of lifetime issues).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm my understanding: you would like to avoid all .clone() calls involved in creating TrieKey from the main thread, is that right?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like an API that encourages one to clone() a large data structure is problematic. So we should think hard about it. In this case, I wonder if changing the function to take AccountId will make the API less "error prone" to use.

runtime/runtime/src/prefetch.rs Outdated Show resolved Hide resolved
@pugachAG pugachAG added the C-housekeeping Category: Refactoring, cleanups, code quality label Dec 13, 2022
@pugachAG pugachAG merged commit 87f0c73 into near:master Dec 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-housekeeping Category: Refactoring, cleanups, code quality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants