upgrade from v4.0.15 to v4.0.16, ticdc stoped with CDC:ErrProcessorUnknown

[seiya-annie]

What did you do?

1. in v4.0.15, set cdc changefeed, write data
2. upgrade to v4.0.16
3. write data
4. scale-out tidb,pd,tikv
5. scale-in tidb/pd/tikv
6. check CDC status

upstream: tidb
downstream: tidb

What did you expect to see?

task works normal

What did you see instead?

task stopped at "2021-12-08 14:08:26.157"
[
{
"id": "simple-cdc-task",
"summary": {
"state": "normal",
"tso": 429639258906820611,
"checkpoint": "2021-12-08 14:08:26.157",
"error": {
"addr": "cdc-peer:8300",
"code": "CDC:ErrProcessorUnknown",
"message": "[CDC:ErrReachMaxTry]reach maximum try: 8"
}
}
}
]
[root@cdc-0 bin]# date
Wed Dec 8 16:09:36 CST 2021
[root@cdc-0 bin]#
cdc.log.gz

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

-------+
| Release Version: v4.0.16
Edition: Community
Git Commit Hash: 75f81d2d7faae72575c4e471ec0b96756796284a
Git Branch: heads/refs/tags/v4.0.16
UTC Build Time: 2021-12-02 04:17:32
GoVersion: go1.13
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false |

TiCDC version (execute cdc version):

[root@cdc-0 bin]# ./cdc version
Release Version: v4.0.16
Git Commit Hash: 6283ec07544ee7011c52aad154bf93a0ee412fa4
Git Branch: heads/refs/tags/v4.0.16
UTC Build Time: 2021-12-02 13:07:10
Go Version: go version go1.13 linux/amd64
Failpoint Build: false

[seiya-annie]

cluster config and metrics, please found in "https://clinic.pingcap.com:4433/diag/files?uuid=fd259311fcd65999-f5b71e9c5d5129e-15c3a73217ffd944"

[liuzix]

Please provide the relevant DDLs and DMLs in the workload.

[seiya-annie]

maybe failed for this case:
https://github.com/pingcap/automated-tests/blob/master/ticases/transaction/tests/ddl/add_column/addcolumn_date_after.t.sql

[VelocityLight]

update