Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test Failure: disaster_recovery_2 #390

Closed
linh2931 opened this issue Jul 22, 2024 · 3 comments · Fixed by #426, #428, #484, #486 or #494
Closed

Test Failure: disaster_recovery_2 #390

linh2931 opened this issue Jul 22, 2024 · 3 comments · Fixed by #426, #428, #484, #486 or #494
Assignees
@heifner
Labels
👍 lgtm OCI Work exclusive to OCI team test-instability tag issues for flaky tests, high priority to address
Milestone
Spring v1.0.0-rc1

Comments

@linh2931
Copy link
Member

https://github.com/AntelopeIO/spring/actions/runs/10048012787/job/27771514855?pr=382#step:4:1595

Traceback (most recent call last):
  File "/__w/spring/spring/build/tests/disaster_recovery_2.py", line 140, in <module>
    assert node.waitForLibToAdvance(), "Node did not advance LIB after relaunch"
@enf-ci-bot enf-ci-bot moved this to Todo in Team Backlog Jul 22, 2024
@heifner
Copy link
Member

heifner commented Jul 23, 2024

The test restarts 5 nodes from a snapshot. The test failed because node0 could not connect to any node when launched so it waited 30 seconds before trying again. During this 30 seconds it happened to be producing blocks. Once it was able to connect it was busy feeding blocks to the other node.

Options:

  • Modify the test to wait longer than the default 30 seconds for LIB to advance. Note the default for reconnect attempts is 30 seconds.
  • Modify net_plugin to attempt connections quicker than 30 seconds at launch and back off to configured connection monitor time after a number of connection attempts.

@arhag arhag added 👍 lgtm test-instability tag issues for flaky tests, high priority to address and removed triage labels Jul 26, 2024
@arhag arhag added this to the Savanna: Production-Ready milestone Jul 26, 2024
@heifner heifner self-assigned this Jul 29, 2024
@heifner heifner added the OCI Work exclusive to OCI team label Jul 29, 2024
@heifner heifner moved this from Todo to In Progress in Team Backlog Jul 29, 2024
heifner added a commit that referenced this issue Jul 29, 2024
@heifner
GH-390 Provide a bit more time for connections to be established on r…
4e3000f 
…estart
@heifner heifner mentioned this issue Jul 29, 2024
Merged
@heifner heifner linked a pull request Jul 29, 2024 that will close this issue
[1.0-beta4] Test: Provide more time for connections to be established #426
Merged
heifner added a commit that referenced this issue Jul 29, 2024
@heifner
Merge pull request #426 from AntelopeIO/GH-390-test-beta4
ece1527 
[1.0-beta4] Test: Provide more time for connections to be established
heifner added a commit that referenced this issue Jul 29, 2024
@heifner
Merge remote-tracking branch 'spring/release/1.0-beta4' into GH-390-t…
231adec 
…est-main
@heifner heifner mentioned this issue Jul 29, 2024
Merged
heifner added a commit that referenced this issue Jul 29, 2024
@heifner
Merge pull request #428 from AntelopeIO/GH-390-test-main
7e6e186 
[1.0-beta4 -> main] Test: Provide more time for connections to be established
@github-project-automation github-project-automation bot moved this from In Progress to Done in Team Backlog Jul 29, 2024
@spoonincode
Copy link
Member

appears to still exist?
https://github.com/AntelopeIO/spring/actions/runs/10272637950/job/28425737014
reopening for now but if felt strongly can open new one instead

@spoonincode spoonincode reopened this Aug 6, 2024
@github-project-automation github-project-automation bot moved this from Done to Todo in Team Backlog Aug 6, 2024
heifner added a commit that referenced this issue Aug 6, 2024
@heifner
GH-390 pending fork block is in first index of known_blocks.ids not t…
dcc789e 
…he last index.
@heifner heifner mentioned this issue Aug 6, 2024
Merged
@heifner heifner linked a pull request Aug 6, 2024 that will close this issue
[1.0-beta4] P2P: Fix handling of known pending block #484
Merged
@heifner heifner mentioned this issue Aug 7, 2024
Merged
heifner added a commit that referenced this issue Aug 7, 2024
@heifner
Merge pull request #484 from AntelopeIO/GH-390-p2p-notice-beta4
62b759b 
[1.0-beta4] P2P: Fix handling of known pending block
heifner added a commit that referenced this issue Aug 7, 2024
@heifner
Merge remote-tracking branch 'spring/release/1.0-beta4' into GH-390-p…
7a55bc2 
…2p-notice-main
@heifner heifner mentioned this issue Aug 7, 2024
Merged
@heifner heifner moved this from Todo to In Progress in Team Backlog Aug 7, 2024
@heifner heifner moved this from In Progress to Awaiting Review in Team Backlog Aug 7, 2024
heifner added a commit that referenced this issue Aug 7, 2024
@heifner
Merge pull request #486 from AntelopeIO/GH-390-p2p-notice-main
01a7cd2 
[1.0-beta4 -> main] P2P: Fix handling of known pending block
@github-project-automation github-project-automation bot moved this from Awaiting Review to Done in Team Backlog Aug 7, 2024
heifner added a commit that referenced this issue Aug 7, 2024
@heifner
GH-390 pending fork block is in first index of known_blocks.ids not t…
1e5892b 
…he last index.
heifner added a commit that referenced this issue Aug 7, 2024
@heifner
Merge pull request #2401 from AntelopeIO/GH-390-p2p-notice-5.0
de19d85 
[5.0] P2P: Fix handling of known pending block
heifner added a commit that referenced this issue Aug 7, 2024
@heifner
Merge remote-tracking branch 'origin/release/5.0' into GH-390-p2p-not…
3681afe 
…ice-merge-main
@heifner heifner mentioned this issue Aug 7, 2024
Merged
heifner added a commit that referenced this issue Aug 7, 2024
@heifner
Merge pull request #494 from AntelopeIO/GH-390-p2p-notice-merge-main
6035f21 
[5.0 -> main] P2P: Fix handling of known pending block
@spoonincode
Copy link
Member

noticed again on
https://github.com/AntelopeIO/spring/actions/runs/10376337986/job/28728623008
though on an UBSAN build which of course goes kinda on the slow side. Should we reopen this one or do a new issue?

@heifner heifner reopened this Aug 13, 2024
@github-project-automation github-project-automation bot moved this from Done to Todo in Team Backlog Aug 13, 2024
@heifner heifner closed this as completed Aug 13, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in Team Backlog Aug 13, 2024
@heifner heifner mentioned this issue Aug 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@heifner heifner

Labels
👍 lgtm OCI Work exclusive to OCI team test-instability tag issues for flaky tests, high priority to address
Projects
Team Backlog
Archived in project
Milestone
Spring v1.0.0-rc1
5 participants
@heifner @arhag @spoonincode @linh2931 @enf-ci-bot