Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

9514: Reduce Java allocations when sending internal node hashes during reconnect #9538

Merged
merged 3 commits into from
Oct 31, 2023

Conversation

artemananiev
Copy link
Member

Fix summary:

  • getChildHashes() method in TeacherTreeView is replaced with another method, writeChildHashes(). This new method is expected to write child hashes to the specified output stream in binary format, identical to how list of Hash objects was previously written
  • In StandardTeacherTreeView this method is implemented using the old getChildHashes() and serializing them with SerializableDataOutputStream.writeSerializableList()
  • In VirtualTeacherTreeView this new method delegates to the underlying data source
  • In VirtualDataSource a new method is added for this purpose, loadAndWriteHash()
  • This new method in MerkleDbDataSource loads hash bytes from disk and serializes them to the output stream, bypassing deserialization to Hash

Testing:

  • Existing virtual map reconnect tests in swirlds-merkle are used
  • I found that these tests use MerkleDb, but the data source is not actually used, all nodes are loaded from virtual node cache. Fixed that
  • Then I had to set hashes RAM to disk threshold to 0 to load hashes from disk, not from data source cache
  • Then I found that one of the reconnect tests take 5 mins to complete because of recent async timeout update. Changed the test to use custom timeout (5s)

Fixes: #9514
Signed-off-by: Artem Ananev artem.ananev@swirldslabs.com

…g reconnect

Signed-off-by: Artem Ananev <artem.ananev@swirldslabs.com>
@github-actions
Copy link

github-actions bot commented Oct 27, 2023

Node: Unit Test Results

    2 259 files      2 259 suites   1h 27m 28s ⏱️
118 232 tests 118 198 ✔️ 34 💤 0
126 491 runs  126 457 ✔️ 34 💤 0

Results for commit 9890754.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Oct 27, 2023

Node: E2E Test Results

    1 files      1 suites   20m 32s ⏱️
310 tests 310 ✔️ 0 💤 0
332 runs  332 ✔️ 0 💤 0

Results for commit 9890754.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Oct 27, 2023

Node: Integration Test Results

280 tests   280 ✔️  32m 0s ⏱️
    5 suites      0 💤
    5 files        0

Results for commit 9890754.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Oct 27, 2023

Node: HAPI Test Results

1 228 tests   748 ✔️  1h 15m 34s ⏱️
   165 suites  480 💤
   165 files        0

Results for commit 9890754.

♻️ This comment has been updated with latest results.

Signed-off-by: Artem Ananev <artem.ananev@swirldslabs.com>
@codecov
Copy link

codecov bot commented Oct 27, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (276a5e1) 65.22% compared to head (6a2fd59) 0.00%.
Report is 17 commits behind head on develop.

❗ Current head 6a2fd59 differs from pull request most recent head 9890754. Consider uploading reports for the commit 9890754 to get more accurate results

Additional details and impacted files
@@              Coverage Diff              @@
##             develop   #9538       +/-   ##
=============================================
- Coverage      65.22%       0   -65.23%     
=============================================
  Files           3257       0     -3257     
  Lines         123995       0   -123995     
  Branches       12812       0    -12812     
=============================================
- Hits           80881       0    -80881     
+ Misses         40062       0    -40062     
+ Partials        3052       0     -3052     

see 3257 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

imalygin
imalygin previously approved these changes Oct 27, 2023
Signed-off-by: Artem Ananev <artem.ananev@swirldslabs.com>
@artemananiev artemananiev merged commit ef8de3d into develop Oct 31, 2023
8 of 13 checks passed
@artemananiev artemananiev deleted the 09514-D-skip-hash-deserialization-reconnects branch October 31, 2023 21:08
@artemananiev artemananiev added this to the v0.45 milestone Oct 31, 2023
imalygin pushed a commit that referenced this pull request Nov 13, 2023
…g reconnect (#9538)

Fixes: #9514
Reviewed-by: Cody Littley <cody@swirldslabs.com>, Ivan Malygin <ivan@swirldslabs.com>, Nathan Klick <nathan@swirldslabs.com>, Oleg Mazurov <oleg.mazurov@swirldslabs.com>, Quan Nguyen <quan@swirldslabs.com>
Signed-off-by: Artem Ananev <artem.ananev@swirldslabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reduce Java allocations when sending internal node hashes during reconnect
6 participants