[air] Move to new storage_path API in tests and examples

[krfricke]

Why are these changes needed?

Following #33463, this PR updates our tests, examples, and docs to use the new storage_path API.

The only locations where we continue to use the local_dir statement are tests where we specify both a local dir and a remote dir. For these tests, we can move to an environment-variable based wrapper in the future.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[xwjiang2010]

Unit test looks good. Can we also run two release tests: 1. one tune scalability and 2. one tune cloud test

[krfricke]

Yes, release tests are here: https://buildkite.com/ray-project/release-tests-pr/builds/34534#_

The worker fault tolerance test failure seems unrelated to the changes but I'll double check.

[justinvyu]

Looks good.

I do have a question about a SyncConfig.upload_dir remainder that is causing some unintended effects:

Experiment checkpoint upload logic currently checks for upload_dir in order to determine whether or not to ignore uploading checkpoint dirs. We should just check for self._remote_checkpoint_dir and remove the need for sync_config in this case:

ray/python/ray/tune/execution/experiment_state.py

Line 237 in a9431c7

if bool(self._sync_config.upload_dir):

Should we fix this in the current PR? Or is there another PR queued up already?

[krfricke]

The test failure seems unrelated - it's a segfault that seems to come from a different source. It also seems to be a flaky test. I've left some debug prints in the test for the next time this failure occurs, but assuming the unit tests pass, we can merge this PR.

[krfricke]

Looks good.

I do have a question about a SyncConfig.upload_dir remainder that is causing some unintended effects:

Experiment checkpoint upload logic currently checks for upload_dir in order to determine whether or not to ignore uploading checkpoint dirs. We should just check for self._remote_checkpoint_dir and remove the need for sync_config in this case:

ray/python/ray/tune/execution/experiment_state.py

Line 237 in a9431c7

if bool(self._sync_config.upload_dir):

Should we fix this in the current PR? Or is there another PR queued up already?

Thanks! Overlooked this one. Fixed

[richardliaw]

stamp docs