[Tune] Add trial_name_creator and trial_dirname_creator to TuneConfig

[justinvyu]

Why are these changes needed?

This is a popular user request for these custom trial name and directory name creator functions to be added back to Tuner. Also, this PR updates doc descriptions for recommendations on including some unique trial attribute in the custom names/dirnames that are returned.

tune_config=tune.TuneConfig(
    trial_name_creator=lambda trial: f"{trial.trainable_name}_{trial.trial_id}",
    trial_dirname_creator=lambda trial: f"{trial.trainable_name}_{trial.trial_id}",
),

Open questions that this PR can also address:

• Is there any additional functionality to be added to these creators?
  • Seems like main use is just removing certain dirname defaults such as the hyperparameter config experiment tag as well as the date. Current functionality is good enough.
• Is it useful to provide the entire Trial object for flexibility, or should only a subset of Trial attributes be passed in for clarity?
  • The input to each of these methods could be changed to just a subset:
  • trial_name_creator=lambda trainable_name, trial_id, config: f"{trainable_name}_{trial_id}"
  • Users do seem to prefer the flexibility of the Trial, though.

Related issue number

Closes #29255

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[krfricke]

This is great, thanks!

@amogkam quick question, do you think we should have this as part of the RunConfig? I know the "trial" concept is mostly relevant for tune, but trainers will also create a trial name and directory, so maybe we want to have this flexibility there as well.

[amogkam]

Yeah I think it can be useful- a relevant user question here https://discuss.ray.io/t/train-when-resuming-training-a-new-trial-directory-is-created-even-when-resuming-from-checkpoint/7672.

But agreed, we would need a better API for this than exposing the Trial directly.

@justinvyu @krfricke does it make sense to specify in the docstring that these args are alpha for now and the API for these are subject to change?

[krfricke]

Let's mark it as alpha, then we can move this to run config once we converged on the API for non-tune runs

[justinvyu]

Update: will just flag as alpha for now.

For the user question though, even if we allow this trial dirname to be configured, the checkpoint config num_to_keep would still not be handled correctly, since Trainer(resume_from_checkpoint) creates a completely new experiment (checkpoint manager state doesn't get loaded).

[krfricke]

The current CI failures are due to an early kick-off mismatch. If you merge master and push to this commit, they should go away automatically