binlog replication: support GTID #521

WangXiangUSTC · 2020-03-06T09:21:46Z

What problem does this PR solve?

syncer unit support GTID

What is changed and how it works?

add new struct Location to save binlog position and gtid set
use Location in syncer unit

Check List

Tests

Unit test
Integration test

Side effects

Breaking backward compatibility

Related changes

Need to be included in the release note

WangXiangUSTC · 2020-03-06T09:22:13Z

/run-all-tests tidb=release-3.0

WangXiangUSTC · 2020-03-07T02:57:54Z

/run-all-tests tidb=release-3.0

WangXiangUSTC · 2020-03-07T07:07:20Z

/run-all-tests tidb=release-3.0

pkg/binlog/position.go

syncer/status.go

syncer/checkpoint.go

syncer/checkpoint_test.go

syncer/sharding_group.go

syncer/sharding-meta/shardmeta.go

syncer/streamer_controller.go

syncer/syncer_test.go

WangXiangUSTC · 2020-03-07T11:01:46Z

/run-all-tests tidb=release-3.0

WangXiangUSTC · 2020-03-07T11:14:45Z

/run-all-tests tidb=release-3.0

WangXiangUSTC · 2020-03-07T12:04:34Z

/run-all-tests tidb=release-3.0

syncer/syncer.go

WangXiangUSTC · 2020-03-09T11:59:41Z

/run-all-tests tidb=release-3.0

WangXiangUSTC · 2020-03-09T12:23:35Z

/run-all-tests tidb=release-3.0

WangXiangUSTC · 2020-03-09T12:39:27Z

@csuzhangxc @lichunzhu PTAL

lichunzhu

Rest LGTM

lichunzhu · 2020-03-09T12:54:10Z

syncer/checkpoint.go

-			if binlog.ComparePosition(pos, minCheckpoint) > 0 {
-				cp.globalPoint = newBinlogPoint(pos, nil, pos, nil)
+			if binlog.CompareLocation(location, binlog.NewLocation(cp.cfg.Flavor)) > 0 {
+				cp.globalPoint = newBinlogPoint(location, location, nil, nil)


Should we create another location to avoid the reuse of GTIDSet?

all addressed in 53bef75

lichunzhu · 2020-03-09T12:55:01Z

syncer/checkpoint.go

@@ -640,7 +659,7 @@ func (cp *RemoteCheckPoint) Load(tctx *tcontext.Context, schemaTracker *schema.T
 			mSchema = make(map[string]*binlogPoint)
 			cp.points[cpSchema] = mSchema
 		}
-		mSchema[cpTable] = newBinlogPoint(pos, &ti, pos, &ti)
+		mSchema[cpTable] = newBinlogPoint(location, location, &ti, &ti)


lichunzhu · 2020-03-09T12:55:33Z

syncer/checkpoint.go

 		if err != nil {
 			return err
 		}
 	case config.ModeIncrement:
 		// load meta from task config
 		if cp.cfg.Meta == nil {
 			cp.logCtx.L().Warn("don't set meta in increment task-mode")
+			location1 := binlog.NewLocation(cp.cfg.Flavor)
+			cp.globalPoint = newBinlogPoint(location1, location1, nil, nil)


lichunzhu · 2020-03-09T12:55:57Z

syncer/checkpoint.go

-	if pos != nil {
-		cp.globalPoint = newBinlogPoint(*pos, nil, *pos, nil)
+	if location != nil {
+		cp.globalPoint = newBinlogPoint(*location, *location, nil, nil)


dm/worker/worker.go

pkg/binlog/position.go

csuzhangxc · 2020-03-09T13:27:18Z

syncer/checkpoint.go

 	cp.Lock()
 	defer cp.Unlock()
-	cp.saveTablePoint(sourceSchema, sourceTable, pos, ti)
+	cp.saveTablePoint(sourceSchema, sourceTable, point, ti)


Do we need to do Clone for the passed in point? it seems some of the callers do the Clone, but some don't. If so, ditto for other exported methods like SaveGlobalPoint.

yes, I haven't unified the usage for Clone, maybe it's better to let the caller to decide whether to allow the function to update the value, if not allowed, can use Clone.
I add a Clone here, maybe we can unified the usage later.

OK, it's fine.

syncer/checkpoint_test.go

syncer/sharding-meta/shardmeta.go

syncer/checkpoint.go

WangXiangUSTC · 2020-03-09T14:12:02Z

/run-all-tests tidb=release-3.0

WangXiangUSTC · 2020-03-09T14:30:25Z

/run-all-tests tidb=release-3.0

WangXiangUSTC · 2020-03-09T14:31:12Z

@lichunzhu @csuzhangxc PTAL again

csuzhangxc

LGTM. 👍

lichunzhu

LGTM

* add new struct Location to save binlog position and gtid set * use Location in syncer unit

WangXiangUSTC added 9 commits March 3, 2020 23:32

use Location to store position and gtid

6f8c040

update

dd80b84

update

80619bf

add TODO

f9ccc8c

use gtid.Set

36b1f8b

update test

4b5ede3

minor fix

0b92af9

enable gtid in test

ce6494d

merge master

93b1609

WangXiangUSTC added the status/WIP This PR is still work in progress label Mar 6, 2020

WangXiangUSTC added 2 commits March 7, 2020 10:31

fix nil point

b705867

update unit test

6b244aa

fix increment test

556a424

csuzhangxc reviewed Mar 7, 2020

View reviewed changes

pkg/binlog/position.go Show resolved Hide resolved

syncer/status.go Outdated Show resolved Hide resolved

syncer/checkpoint.go Outdated Show resolved Hide resolved

syncer/checkpoint.go Outdated Show resolved Hide resolved

csuzhangxc reviewed Mar 7, 2020

View reviewed changes

syncer/checkpoint.go Outdated Show resolved Hide resolved

csuzhangxc reviewed Mar 7, 2020

View reviewed changes

syncer/checkpoint.go Outdated Show resolved Hide resolved

syncer/checkpoint.go Outdated Show resolved Hide resolved

syncer/checkpoint.go Outdated Show resolved Hide resolved

syncer/checkpoint.go Outdated Show resolved Hide resolved

csuzhangxc reviewed Mar 7, 2020

View reviewed changes

syncer/checkpoint.go Outdated Show resolved Hide resolved

syncer/checkpoint_test.go Outdated Show resolved Hide resolved

csuzhangxc reviewed Mar 7, 2020

View reviewed changes

WangXiangUSTC added 2 commits March 7, 2020 18:52

minor fix on pointer

1ac97e0

remove useless code

b5a50a3

minor update

bd70d96

fix shard meta data restore

79d86e6

csuzhangxc reviewed Mar 7, 2020

View reviewed changes

syncer/syncer.go Outdated Show resolved Hide resolved

syncer/syncer.go Outdated Show resolved Hide resolved

syncer/syncer.go Outdated Show resolved Hide resolved

address comment

6d3ab39

Merge branch 'master' into xiang/support_GTID

c63832e

lichunzhu reviewed Mar 9, 2020

View reviewed changes

add location clone

53bef75

csuzhangxc reviewed Mar 9, 2020

View reviewed changes

syncer/checkpoint_test.go Show resolved Hide resolved

syncer/sharding-meta/shardmeta.go Outdated Show resolved Hide resolved

syncer/sharding-meta/shardmeta.go Outdated Show resolved Hide resolved

WangXiangUSTC added 2 commits March 9, 2020 21:53

address comment

5605ab8

address comemnt

35248aa

csuzhangxc reviewed Mar 9, 2020

View reviewed changes

syncer/checkpoint.go Show resolved Hide resolved

address comment

e0b6cb5

add wait time

ca671b5

csuzhangxc reviewed Mar 9, 2020

View reviewed changes

csuzhangxc added priority/important Major change, requires approval from ≥2 primary reviewers status/LGT1 One reviewer already commented LGTM and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Mar 9, 2020

lichunzhu approved these changes Mar 9, 2020

View reviewed changes

lichunzhu added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Mar 9, 2020

lichunzhu merged commit 8ff04d2 into master Mar 9, 2020

csuzhangxc mentioned this pull request Mar 13, 2020

syncer unit need to support GTID #511

Closed

WangXiangUSTC deleted the xiang/support_GTID branch March 16, 2020 04:05

lichunzhu pushed a commit to lichunzhu/dm that referenced this pull request Apr 6, 2020

binlog replication: support GTID (pingcap#521)

aca7850

* add new struct Location to save binlog position and gtid set * use Location in syncer unit

lichunzhu pushed a commit to lichunzhu/dm that referenced this pull request Apr 6, 2020

binlog replication: support GTID (pingcap#521)

7537066

* add new struct Location to save binlog position and gtid set * use Location in syncer unit

lichunzhu pushed a commit to lichunzhu/dm that referenced this pull request Apr 7, 2020

binlog replication: support GTID (pingcap#521)

ea5c829

* add new struct Location to save binlog position and gtid set * use Location in syncer unit

csuzhangxc mentioned this pull request Apr 30, 2020

syncer does not support gtid yet #241

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

binlog replication: support GTID #521

binlog replication: support GTID #521

WangXiangUSTC commented Mar 6, 2020 •

edited

Loading

WangXiangUSTC commented Mar 6, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 9, 2020

WangXiangUSTC commented Mar 9, 2020

WangXiangUSTC commented Mar 9, 2020

lichunzhu left a comment

lichunzhu Mar 9, 2020

WangXiangUSTC Mar 9, 2020

lichunzhu Mar 9, 2020

lichunzhu Mar 9, 2020

lichunzhu Mar 9, 2020

csuzhangxc Mar 9, 2020

WangXiangUSTC Mar 9, 2020

csuzhangxc Mar 9, 2020

WangXiangUSTC commented Mar 9, 2020

WangXiangUSTC commented Mar 9, 2020

WangXiangUSTC commented Mar 9, 2020

csuzhangxc left a comment

lichunzhu left a comment

binlog replication: support GTID #521

binlog replication: support GTID #521

Conversation

WangXiangUSTC commented Mar 6, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

WangXiangUSTC commented Mar 6, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 7, 2020

WangXiangUSTC commented Mar 9, 2020

WangXiangUSTC commented Mar 9, 2020

WangXiangUSTC commented Mar 9, 2020

lichunzhu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WangXiangUSTC commented Mar 9, 2020

WangXiangUSTC commented Mar 9, 2020

WangXiangUSTC commented Mar 9, 2020

csuzhangxc left a comment

Choose a reason for hiding this comment

lichunzhu left a comment

Choose a reason for hiding this comment

WangXiangUSTC commented Mar 6, 2020 •

edited

Loading