[core][state] Task state fault tolerance - GCS marks children task failed if parent task fails. [1/x]

[rickyyx]

Signed-off-by: rickyyx rickyx@anyscale.com

Why are these changes needed?

We currently don't mark children tasks as failed if the parent fails. This PR adds a synchronous routine MarkTaskTreeFailed for each newly reported task event. And marks all children tasks in the task tree, with the reported task as the root, as failed, if:

1. The reported task itself is a task failure.
2. The parent of the reported task has already failed.

In the next PRs:

• When a job failed, all tasks and their children with the failed job id will be marked as failed.
• Detached actor's tasks should not be failed if their parents fail.
• We will add a column to TaskState for task exit details, e.g.exit_detail = Task failed because of parent task XXX failed at timestamp YYY.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ericl]

@rkooo567 , could you do the main review?

[rkooo567]

@rickyyx lmk when it is ready to review again (plz remove the label!)

[rkooo567]

I am going to have another round of review (haven't reviewed child/parent update part and unit tests)!

[rickyyx]

Running release tests: https://buildkite.com/ray-project/release-tests-pr/builds/25388

Also see unresolved comments for follow-ups.

[rickyyx]

No regression from runs: https://buildkite.com/ray-project/release-tests-pr/builds/25408 , many_tasks is failing in master.
Other CI tests are not relevant or already fixed in master.
cc @rkooo567