ticdc redo apply shouldn't depend on meta

[hicqu]

Before asking a question, make sure you have

• Searched existing Stack Overflow questions
• Googled your question
• Searched open and closed GitHub issues
• Read the documentation

What is your question?

In the current implement, when applying redo log on a given downstream sink, the event timestamp range is specified from (meta.checkpointTs, meta.resolvedTs].

redo-log meta file is flushed periodically. However, events can be flushed into downstream sinks after log file is flushed instead of redo-log meta. It makes cdc redo apply not be able to ensure downstream integration. For example:

1. cdc table redo log is flushed with timestamp ts1;
2. however the current redo-meta resolved timestamp is ts0, which is less than ts1;
3. events before ts1 have been flushed into downstream sinks;
4. cdc is killed and cdc redo apply is called with the current redo-meta.
5. after recovered, the downstream can contain events after the point ts0, which breaks the required integration.

I have prepared a branch, and please notice the fatal log. And the panic really happens when I run the branch:

[2022/07/05 16:00:27.117 +08:00] [FATAL] [mysql.go:237] ["[QP] mysqlSink.FlushRowChangedEvents is called with ts less than LogWriterResolved"] [stack="github.com/pingcap/tiflow/cdc/sink/mysql.(*mysqlSink).FlushRowChangedEvents\n\tgithub.com/pingcap/tiflow/cdc/sink/mysql/mysql.go:237\ngithub.com/pingcap/tiflow/cdc/sink.(*tableSink).flushResolvedTs\n\tgithub.com/pingcap/tiflow/cdc/sink/table_sink.go:109\ngithub.com/pingcap/tiflow/cdc/sink.(*tableSink).FlushRowChangedEvents\n\tgithub.com/pingcap/tiflow/cdc/sink/table_sink.go:86\ngithub.com/pingcap/tiflow/cdc/processor/pipeline.(*sinkNode).flushSink\n\tgithub.com/pingcap/tiflow/cdc/processor/pipeline/sink.go:155\ngithub.com/pingcap/tiflow/cdc/processor/pipeline.(*sinkNode).updateBarrierTs\n\tgithub.com/pingcap/tiflow/cdc/processor/pipeline/sink.go:338\ngithub.com/pingcap/tiflow/cdc/processor/pipeline.(*sinkNode).HandleMessage\n\tgithub.com/pingcap/tiflow/cdc/processor/pipeline/sink.go:329\ngithub.com/pingcap/tiflow/cdc/processor/pipeline.(*sinkNode).Receive\n\tgithub.com/pingcap/tiflow/cdc/processor/pipeline/sink.go:289\ngithub.com/pingcap/tiflow/pkg/pipeline.(*nodeRunner).run\n\tgithub.com/pingcap/tiflow/pkg/pipeline/runner.go:65\ngithub.com/pingcap/tiflow/pkg/pipeline.(*Pipeline).driveRunner\n\tgithub.com/pingcap/tiflow/pkg/pipeline/pipeline.go:97"]

The test shows that the above guess is correct.

To resolve the critical bug, we can consider to

• always flush log meta when flushing log files, or
• remove log meta from the design, and put checkpointTs and resolvedTs into redo-log entry

[hicqu]

/label bug-from-internal-test

[amyangfei]

The timing should be

1. flush redo log meta
2. -> update global resolved ts
3. -> flush row changes to backend sink according to global resolved ts

Which rule above is broken?

[amyangfei]

/cc @CharlesCheung96

[hicqu]

/label

[CharlesCheung96]

/type bug

[CharlesCheung96]

/label affects-5.3
/label affects-5.4
/label affects-6.1

[hicqu]

/label severity/critical

[ti-chi-bot]

@hicqu: The label(s) severity/critical cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, affects-4.0, affects-5.0, affects-5.1, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, may-affects-4.0, may-affects-5.0, may-affects-5.1, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, question, release-blocker, wontfix, require-LGT1, require-LGT3.

In response to this:

/label severity/critical

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[zhaoxinyu]

/severity critical

[zhaoxinyu]

/remove may-affects-4.0

[zhaoxinyu]

/remove-label may-affects-4.0

[zhaoxinyu]

/remove-label may-affects-5.0
/remove-label may-affects-5.1
/remove-label may-affects-5.2

[zhaoxinyu]

/remove-label may-affects-6.0

[nongfushanquan]

/label affects-6.1

[nongfushanquan]

/open

[nongfushanquan]

/reopen

[ti-chi-bot]

@nongfushanquan: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nongfushanquan]

after pr 6207, issue 6277 will track the new problem on this situation.

[nongfushanquan]

/close

[ti-chi-bot]

@nongfushanquan: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nongfushanquan]

/area ticdc