changefeed (ticdc): remove error state and add pending, warning state (#9358)

[ti-chi-bot]

This is an cherry-pick of #9358 #9687 #9593

What problem does this PR solve?

Issue Number: ref #9273, ref #9697, ref #9594

What is changed and how it works?

After this PR, the error status of changefeed will be removed, and the warning status will be added.
The changefeed status transition is shown in the following figure.

1. The user pauses changefeed, and changefeed enters the stopped state.
2. The user resumes changefeed, and changefeed returns to the normal state.
3. Changefeed encounters an error and enters the warning state. Changefeed automatically retries in the warning state.
4. If the changefeed retry is successful and the CheckpointTs advances beyond the CheckpointTs when the error occurred, the changefeed enters the normal state.
5. If the changefeed error retry exceeds the maximum retry time limit, it enters the failed state. After entering the failed state, the changefeed will block the GC of the upstream TiDB cluster. The default blocking time is gc-ttl. Users can manually recover the changefeed within the gc-ttl time.
6. If the changefeed encounters an unrecoverable error during the running process, it directly enters the failed state.
7. The changefeed is complete, and the its progress has reached the preset TargetTs.
8. If the changefeed is in the stopped state for a time exceeding the duration specified by gc-ttl, it cannot be recovered.

Check List

Tests

• Unit test
• Integration test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

None

[ti-chi-bot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nongfushanquan]

/retest

[asddongmen]

/test all

[asddongmen]

/test all

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [asddongmen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nongfushanquan]

/retest

[asddongmen]

/test dm-integration-test

[asddongmen]

/test dm-integration-test