[core] Graceful shutdown in TaskEventBuffer destructor

[rickyyx]

Why are these changes needed?

When the core worker quick exits, there's a chance that the additional thread owned by the TaskEventBufferImpl hasn't been joined yet, resulting in errors in the stacktrace.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rkooo567]

  {
    absl::MutexLock lock(&mutex_);
    // It's now safe to disconnect the GCS client since it will not be used by any
    // callbacks.
    gcs_client_->Disconnect();
  }

This part from Stop(), is this still valid? Maybe you should check if this is nullptr or not

[rkooo567]

Also what's the error message when this issue comes up? I am a little worried this won't actually fix the issue depending on the destructor order.

Maybe it is better adding shutdown flag and make periodic job no-op when shutdown == true?

[rkooo567]

or have you verified this fixes the issue manually?

[rickyyx]

or have you verified this fixes the issue manually?

Yes - i verified by running some workload (microbenchmars) and see the error not showing up.