[Tune] Cancel pg.ready() task for pending trials that end up reusing an actor

[justinvyu]

Why are these changes needed?

This PR cancels pg.ready() tasks that are pending node assignment forever, in the case that trials are reusing actors. For Tune experiments with many trials, this clutters the user's dashboard with a bunch of these pg.ready() tasks that don't go away. The trials corresponding to these requests have already been assigned different actors, so there is no point keeping the futures around.

Context

Currently, the PlacementGroupResourceManager uses the pg.ready() future to determine when a placement group is ready for an actor to be created with it.

Let's say we're on a cluster with 10 CPUs, each trial needs 2 CPUs, and we want to run 100 trials. Tune will schedule 5 actors, and Tune will also have some number of pending trials that also try to allocate a placement group (the default is 16 pending in this case I believe, see the TUNE_MAX_PENDING_TRIALS_PG env var).

In the beginning, the state will be:

• 5 trials with 2 CPUs each.
• 16 trials, associated with 16 pending pg.ready() futures.

Then, when the first trial finishes, one of the 16 pending trials will end up reusing the actor of the finished trial.
The resources will not be relinquished from that actor, so the new trial's pg.ready() will never finish, and we've also lost the reference.

@jjyao Question: Why isn't it garbage collected automatically? I think it's because pg.ready() calls some global function that always has at least 1 reference: bundle_reservation_check_func.

So, the pending futures will just keep accumulating as trials keep reusing actors. This leads to the dashboard looking like:

After this change, there will only ever be ~16 pending futures, which are just the PENDING trials that will eventually reuse an actor and cancel the future.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[krfricke]

Thanks!

[jjyao]

Question: Why isn't it garbage collected automatically?

Ray currently doesn't automatically cancel the task when the return object is GCed to support fire and forget.