*: chose one index to output delete events #787

zier-one · 2020-07-22T10:06:27Z

What problem does this PR solve?

fix the #728

What is changed and how it works?

we will select one unique index(including primary key) as the handle index for every table
the delete events are generated based on the handle index
the column of handle index will be marked in column flag
if a unique index includes a virtual generated column, we regard the index as invalid.
the stored generated columns are captured now and it will be also marked in column flag
provides a new dispatcher named index-value
the rwoid dispatcher is instead by index-value
the where_handle field in Column is deprecation

Check List

Tests

Unit test
Integration test

Release note

select one unique index as the handle index to generate the delete events

zier-one · 2020-07-22T10:06:41Z

/run-all-tests

…ete_from_one_index

zier-one · 2020-07-22T11:01:51Z

/run-all-tests

zier-one · 2020-07-24T06:49:24Z

/run-all-tests

zier-one · 2020-07-24T07:03:17Z

/run-all-tests

…ete_from_one_index

zier-one · 2020-07-24T07:56:10Z

/run-all-tests

codecov-commenter · 2020-07-24T08:10:08Z

Codecov Report

Merging #787 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master       #787   +/-   ##
===========================================
  Coverage   31.7860%   31.7860%           
===========================================
  Files            98         98           
  Lines         11181      11181           
===========================================
  Hits           3554       3554           
  Misses         7278       7278           
  Partials        349        349

zier-one · 2020-07-27T03:06:14Z

/run-all-tests

amyangfei · 2020-07-28T04:21:50Z

Please add a release note

zier-one · 2020-07-28T06:17:15Z

Please add a release note

done

cdc/model/sink.go

cdc/entry/mounter.go

cdc/entry/schema_storage.go

amyangfei · 2020-07-28T09:25:19Z

cdc/entry/schema_storage.go

+	}
+	if handleIndexOffset >= 0 {
+		ti.HandleIndexID = ti.Indices[handleIndexOffset].ID
+	}


Is it better to add fatal log here, if handle index is not found, we can record more information here.

I check it here:
https://github.com/pingcap/ticdc/pull/787/files#diff-8cae4fdc1c5f0a4e944025f65917425fR437

But the check in mounter can't log enough debug information

cdc/sink/dispatcher/table.go

cdc/entry/schema_storage.go

amyangfei · 2020-07-29T07:28:59Z

/run-kafka-tests

zier-one · 2020-07-29T08:35:57Z

/run-all-tests

amyangfei · 2020-07-29T09:36:11Z

/run-all-tests tidb=release-4.0 tikv=release-4.0 pd=release-4.0

amyangfei · 2020-07-29T10:38:45Z

/run-kafka-tests tidb=release-4.0 tikv=release-4.0 pd=release-4.0

# Conflicts: # cdc/entry/mounter.go # cdc/entry/schema_storage.go

zier-one · 2020-08-03T05:38:42Z

/run-all-tests

# Conflicts: # cdc/entry/mounter.go # cdc/sink/dispatcher/default.go

zier-one · 2020-08-03T07:23:46Z

/run-all-tests

zier-one · 2020-08-03T07:55:08Z

/run-all-tests

zier-one · 2020-08-03T08:03:27Z

/run-all-tests

liuzix

The rest lgtm

cdc/model/sink.go

cdc/entry/mounter.go

cdc/model/schema_storage.go

cdc/model/sink.go

Co-authored-by: Zixiong Liu <zl2683@columbia.edu>

zier-one · 2020-08-03T09:32:48Z

The rest lgtm

thanks a lot

liuzix · 2020-08-03T09:41:17Z

/lgtm

zier-one · 2020-08-03T09:42:48Z

/merge

ti-srebot · 2020-08-03T09:42:55Z

Your auto merge job has been accepted, waiting for:

815
797

ti-srebot · 2020-08-03T10:15:02Z

/run-all-tests

chose one index to output delete events

476fca4

leoppro added 4 commits July 22, 2020 18:11

Merge branch 'master' into delete_from_one_index

253f351

using handleKey

483b1b3

Merge remote-tracking branch 'pingcap/master' into delete_from_one_index

96be841

Merge branch 'delete_from_one_index' of github:leoppro/ticdc into del…

613a495

…ete_from_one_index

fix the bug about generate column

6be0845

leoppro added 2 commits July 24, 2020 14:54

fix check

b6d95a2

fix test

67975bb

zier-one added the status/ptal Could you please take a look? label Jul 24, 2020

zier-one marked this pull request as ready for review July 24, 2020 07:10

leoppro added 3 commits July 24, 2020 15:10

Merge branch 'master' into delete_from_one_index

1c6e2fb

update comment

47cbf5c

Merge branch 'delete_from_one_index' of github:leoppro/ticdc into del…

265fbe9

…ete_from_one_index

zier-one changed the title ~~chose one index to output delete events~~ *: chose one index to output delete events Jul 24, 2020

leoppro added 2 commits July 27, 2020 11:02

Merge branch 'master' into delete_from_one_index

7f29d73

fmt

63abc43

zier-one mentioned this pull request Jul 27, 2020

mounter: support more column flag #796

Merged

amyangfei added this to the v4.0.4 milestone Jul 28, 2020

amyangfei reviewed Jul 28, 2020

View reviewed changes

cdc/entry/schema_storage.go Outdated Show resolved Hide resolved

update

46ad655

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 29, 2020

leoppro added 2 commits August 3, 2020 12:27

Merge remote-tracking branch 'pingcap/master' into delete_from_one_index

5b0ccf8

Merge remote-tracking branch 'pingcap/master' into delete_from_one_index

d2dcbb4

# Conflicts: # cdc/entry/mounter.go # cdc/entry/schema_storage.go

zier-one added the release-blocker This issue blocks a release. Please solve it ASAP. label Aug 3, 2020

Merge remote-tracking branch 'pingcap/master' into delete_from_one_index

4c22abf

# Conflicts: # cdc/entry/mounter.go # cdc/sink/dispatcher/default.go

add debug code

9147906

fix bug

9627608

liuzix reviewed Aug 3, 2020

View reviewed changes

Apply suggestions from code review

0f8623f

Co-authored-by: Zixiong Liu <zl2683@columbia.edu>

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Aug 3, 2020

ti-srebot approved these changes Aug 3, 2020

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Aug 3, 2020

ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 3, 2020

Merge branch 'master' into delete_from_one_index

e3c241e

ti-srebot merged commit aa496b9 into pingcap:master Aug 3, 2020

zier-one deleted the delete_from_one_index branch September 14, 2020 03:27

Rustin170506 mentioned this pull request Jul 15, 2021

puller,mounter,processor: always pull the old value internally #2271

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: chose one index to output delete events #787

*: chose one index to output delete events #787

zier-one commented Jul 22, 2020 •

edited

Loading

zier-one commented Jul 22, 2020

zier-one commented Jul 22, 2020

zier-one commented Jul 24, 2020

zier-one commented Jul 24, 2020

zier-one commented Jul 24, 2020

codecov-commenter commented Jul 24, 2020 •

edited

Loading

zier-one commented Jul 27, 2020

amyangfei commented Jul 28, 2020

zier-one commented Jul 28, 2020

amyangfei Jul 28, 2020

zier-one Jul 28, 2020

amyangfei Jul 29, 2020

amyangfei commented Jul 29, 2020

zier-one commented Jul 29, 2020

amyangfei commented Jul 29, 2020

amyangfei commented Jul 29, 2020

zier-one commented Aug 3, 2020

zier-one commented Aug 3, 2020

zier-one commented Aug 3, 2020

zier-one commented Aug 3, 2020

liuzix left a comment

zier-one commented Aug 3, 2020

liuzix commented Aug 3, 2020

zier-one commented Aug 3, 2020

ti-srebot commented Aug 3, 2020

ti-srebot commented Aug 3, 2020

*: chose one index to output delete events #787

*: chose one index to output delete events #787

Conversation

zier-one commented Jul 22, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

zier-one commented Jul 22, 2020

zier-one commented Jul 22, 2020

zier-one commented Jul 24, 2020

zier-one commented Jul 24, 2020

zier-one commented Jul 24, 2020

codecov-commenter commented Jul 24, 2020 • edited Loading

Codecov Report

zier-one commented Jul 27, 2020

amyangfei commented Jul 28, 2020

zier-one commented Jul 28, 2020

amyangfei Jul 28, 2020

Choose a reason for hiding this comment

zier-one Jul 28, 2020

Choose a reason for hiding this comment

amyangfei Jul 29, 2020

Choose a reason for hiding this comment

amyangfei commented Jul 29, 2020

zier-one commented Jul 29, 2020

amyangfei commented Jul 29, 2020

amyangfei commented Jul 29, 2020

zier-one commented Aug 3, 2020

zier-one commented Aug 3, 2020

zier-one commented Aug 3, 2020

zier-one commented Aug 3, 2020

liuzix left a comment

Choose a reason for hiding this comment

zier-one commented Aug 3, 2020

liuzix commented Aug 3, 2020

zier-one commented Aug 3, 2020

ti-srebot commented Aug 3, 2020

ti-srebot commented Aug 3, 2020

zier-one commented Jul 22, 2020 •

edited

Loading

codecov-commenter commented Jul 24, 2020 •

edited

Loading