[Serve] Return error string from deploy_serve_application task

[shrekris-anyscale]

Why are these changes needed?

The controller runs the deploy_serve_application task to build and run the user's Serve app. If the task raises an error, the controller will try to deserialize it when it calls ray.get() on the task's reference. If the error contains a custom dependency, the deserialization will fail, and the controller will log an error about the deserialization failing instead of the actual error itself.

This change catches any error in the deploy_serve_application task itself and returns it as a string to the controller. The controller then simply logs the string.

Related issue number

Closes #35677 and #35678.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
    • This change adds a new test to test_controller.py.

[edoakes]

why are these delayed imports here btw?

[shrekris-anyscale]

We can probably move call_app_builder_with_args_if_necessary out of the task, but moving the others would cause the controller to depend on api.py (to access serve.run() and serve.build()) which seems circular.

[edoakes]

let's use a local file instead please to avoid an external dependency

you can import from this file as from ray.serve.tests.test_controller import my_app

you can check it isn't serialized by raising an exception in __reduce__ or by including another non-serializable object as an attribute (and also may as well verify it isn't serializable using ray.cloudpickle in the test case)

[shrekris-anyscale]

Sounds good, I'll move it to the Ray repo. I can't put it in test_controller.py though, since I need to raise the Exception in the file itself (outside the deployment), in order for the error to get propagated back to the user.

I'll create a file in test_config_files and import it from there.

[edoakes]

lgtm, lint failing

[shrekris-anyscale]

The test failures are unrelated:

• Windows Build & Test (test_object_store_metrics) doesn't rely on Serve. It's slightly flaky on master.

• Serve tests (test_standalone3) is flaky on master. I retried the test and it succeeded.

[shrekris-anyscale]

@edoakes This PR is almost ready to merge. The test_object_store_metrics failure looks a bit suspicious to me since it's not very flaky. I'm rerunning it. We can merge it afterwards.

[shrekris-anyscale]

The retry succeeded. @edoakes @architkulkarni This change is ready to merge.