[ci][byod] migrate all data tests to byod

[can-anyscale]

Why are these changes needed?

Release test infra now support running tests using anyscale byod, which is a more reliable way of testing.

Migrate all/most data tests to byod

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• Testing Strategy
  • Unit tests
  • Release tests - https://buildkite.com/ray-project/release-tests-pr/builds/42056 (release tests that fail are already fail on master)

[can-anyscale]

On CI, release tests that fail are already fail on master

[ollie-iterators]

If byod is disabled on pull requests, how are you testing moving data tests to byod?

[aslonnie]

If byod is disabled on pull requests, how are you testing moving data tests to byod?

we have means to run tests with wheels and containers that are already built.

[aslonnie]

via Rota 29 on GIPHY

[matthewdeng]

Can someone from Data team review this? @amogkam @bveeramani @raulchen @scottjlee

[can-anyscale]

data folks, @amogkam @bveeramani @raulchen @scottjlee, if you all don't have a strong opinion, I'll merge this PR in sometimes tomorrow on the merit that tests pass after this change. Thankkks

[scottjlee]

Thanks! qq, I saw these tests with the label team: data did not have the byod config. Are these to be excluded? (e.g. for being chaos / unstable tests)

pipelined_training_50_gb
dataset_shuffle_random_shuffle_1tb
dataset_shuffle_push_based_random_shuffle_100tb
chaos_dask_on_ray_large_scale_test_no_spilling
chaos_dask_on_ray_large_scale_test_spilling
chaos_pipelined_ingestion_1500_gb_15_windows
chaos_dataset_shuffle_push_based_sort_1tb

[can-anyscale]

@scottjlee: thanks for reviewing Scott. Yesss, some of the tests are currently failing on master so I exclude them until they get fixed first