[data] Unit-test physical execution of Datasets

[stephanie-wang]

Description

We should unit-test Datasets' interaction with the Ray core layer by tracing the calls to f.remote() and checking that these match the expected behavior. This could help prevent performance regressions like the following:

• [data] Dataset.schema() may get recomputed each time #37077 - check that certain Dataset calls do not trigger execution
• [data] Use default scheduling strategy to fix failing dataset_shuffle_push_based_sort_1tb test #36722 - check that tasks are getting submitted with the correct task .options (e.g., scheduling strategy)

Ideally all (performance-critical) Dataset tests should explicitly say what Ray core tasks they expect to produce.

Use case

No response

[raulchen]

The tracking code can be added to python/ray/data/_internal/remote_fn.py::cached_remote_fn

[anyscalesam]

Relevant issues that you described are closed @stephanie-wang; can we close this issue?