[tune] Turn on new execution path per default [no_early_kickoff]

[krfricke]

Why are these changes needed?

We've introduced a new execution path for Ray Tune behind a feature flag. The unit tests have been stable for some time. To get better dogfooding results, we should enable it per default, so that all unit and release tests are run using the new engine.

If we run into problems, we can easily revert this PR before we cut the 2.5 release branch.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[pcmoritz]

Hey Kai, are there any API changes as part of the new execution path? If no that's perfect, if yes we should make sure no existing code breaks as part of the change :)

[krfricke]

No API changes! Except for not accepting the already deprecated RayTrialExecutor anymore:

    elif trial_executor:
        warnings.warn(
            "Passing a custom `trial_executor` is deprecated and will be removed "
            "in the future.",
            DeprecationWarning,
        )

Otherwise - ideally - people won't even notice.

[krfricke]

With some minor changes (moving 2 code blocks and one safety check) all CI tests are passing and failing tests seem unrelated.

I've also kicked off a round of release tests (all tune_ tests and the air_benchmark_ tests), all of which are passing (except for 2 unstable tests that also fail on master).

Thus it looks like we have functional parity and can go ahead.

I'm preparing a PR to port over unit tests (#34833) which hopefully will be ready this week, too.