Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[3.2] Fix race condition on trace_api_plugin shutdown #591

Merged
merged 1 commit into from
Jul 2, 2022

Conversation

heifner
Copy link
Member

@heifner heifner commented Jul 2, 2022

Failure https://github.com/eosnetworkfoundation/mandel/runs/7143297502?check_suite_focus=true pointed to an issue in shutdown of the trace_api_plugin.

slice_directory::stop_maintenance_thread() sets atomic bool _maintenance_shutdown = true and calls notify_one() on the _maintenance_condition condition variable. This is a race condition because it doesn't acquire the condition variable mutex setting up the possibility that _maintenance_shutdown can be set to true and notify_one called after the while check in slice_directory::start_maintenance_thread but before the wait() causing it to wait forever. This then blocks the slice_directory::stop_maintenance_thread() call of join() on the main thread, blocking the shutdown of all other plugins.

Changed the logic to correctly use a mutex and condition variable around _best_known_lib & _maintenance_shutdown. Also made these two variables non-atomic since they are now only accessed via the mutex. Also release the mutex before running the run_maintenance_tasks() which appeared to be the original intention with the use of the local variables.

@heifner heifner added OCI OCI working this issue... 3.1 RC2 labels Jul 2, 2022
@heifner heifner merged commit 6e17696 into main Jul 2, 2022
@heifner heifner deleted the GH-590-shutdown-race branch July 2, 2022 18:05
@heifner heifner changed the title [3.1] Fix race condition on trace_api_plugin shutdown [3.2] Fix race condition on trace_api_plugin shutdown Jul 2, 2022
@heifner heifner removed the 3.1 RC2 label Jul 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OCI OCI working this issue...
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants