Implemented PITR-2 #6408

arindamnayak · 2020-07-03T18:47:24Z

Implemented PITR-2
Fixes #6267, fixes #4886

Here are the changes I did.

vttablet will accept binlog server parameter
all those parameters will be used while restoring the tablet and it need a snapshot keyspace.
from the snapshot keyspace type, we will get the snapshot_time
we will get the GTID from the snapshot time
then we will replicate from binlog server upto that GTID
once we reach that GTID , the slave connection will be closed and reset.

- Added flags for restore to time - Added flags for binlog server details Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

- added ability to fetch the GTID from the restore time - added ability to apply the binlogs till the above GTID Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

…chan Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

go/vt/vttablet/tabletmanager/restore.go

arindamnayak · 2020-07-14T17:41:01Z

go/vt/vttablet/tabletserver/schema/engine.go

@@ -64,6 +64,11 @@ type Engine struct {
 	notifierMu  sync.Mutex
 	notifiers   map[string]notifier

+	// SkipMetaCheck skips the metadata about the database and table information
+	SkipMetaCheck bool


@sougou , is it ok if we add this flag here? This flag will skip the sql query to get table information ( form information_schema). We need this when we need to get the GTID from timestamp , as the binlog server does not support the query and we also don't need that info in that process.

Sounds good for now. I see that you're just using it to covert timestamp to gtid. There may be a simpler way to do that without having to use vstream, but we can use this method until we figure that out.

…le names Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi

Mostly comments and documentation, but there is an important logic issue as well.

.github/workflows/docker_test_2.yml

go/mysql/flavor_mysql.go

go/test/endtoend/recovery/pitr/pitr_test.go

deepthi · 2020-07-16T01:45:43Z

go/vt/vttablet/tabletmanager/restore.go

+	binlogPort           = flag.Int("binlog_port", 0, "(PITR restore parameter) port of binlog server.")
+	binlogUser           = flag.String("binlog_user", "", "(PITR restore parameter) username of binlog server.")
+	binlogPwd            = flag.String("binlog_password", "", "(PITR restore parameter) password of binlog server.")
+	timeoutForGTIDLookup = flag.Duration("binlog_lookup_timeout", 60*time.Second, "(PITR restore parameter) timeout for fetching gtid from timestamp.")


what do you think of gtid_lookup_timeout for this flag name?

that does not signifies which gtid we are referring to, let me know if we should use that.

recovery_gtid_lookup_timeout?

go/vt/vttablet/tabletmanager/restore.go

go/vt/vttablet/tabletmanager/vreplication/replica_connector.go

go/vt/vttablet/tabletserver/schema/engine.go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak · 2020-07-16T14:43:00Z

Mostly comments and documentation, but there is an important logic issue as well.

I have implemented the suggestion you mentioned.

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

aquarapid · 2020-07-17T16:54:12Z

go/mysql/flavor_mysql.go

@@ -32,7 +32,8 @@ type mysqlFlavor struct{}

 // masterGTIDSet is part of the Flavor interface.
 func (mysqlFlavor) masterGTIDSet(c *Conn) (GTIDSet, error) {
-	qr, err := c.ExecuteFetch("SELECT @@GLOBAL.gtid_executed", 1, false)
+	// making @@global as lowercase, as the PITR depends on binlog server, which honours only lowercase `global` value


Let's just make this:

// keep @@global as lowercase, as some servers like the Ripple binlog server only honors a lowercase `global` value

Otherwise people might get confused (since this is not PITR-specific code)

Agreed, I have updated the comment.

deepthi · 2020-07-17T18:20:41Z

Can you balance tests between docker_test_1 and docker_test_2?

docker test 2 / Docker Test 2 (pull_request) Successful in 19m
docker_test_1 / Docker Test 1 (pull_request) Successful in 9m

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak · 2020-07-18T12:21:46Z

Can you balance tests between docker_test_1 and docker_test_2?
docker test 2 / Docker Test 2 (pull_request) Successful in 19m
docker_test_1 / Docker Test 1 (pull_request) Successful in 9m

I have balanced the tests

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi

The new logic looks good.
A few more nits. We can merge once those are addressed.

go/test/endtoend/cluster/cluster_util.go

go/test/endtoend/recovery/pitr/main_test.go

deepthi · 2020-07-18T16:29:18Z

go/test/endtoend/recovery/pitr/main_test.go

+		if proc, err := tablet.MysqlctlProcess.StartProcess(); err != nil {
+			return err
+		} else {
+			// ignore golint warning, we need the else block to use proc


don't you need a //nolint here? otherwise CI will fail.

not sure why golangci-lint did not capture this earlier, it is passing without //nolint, to be on safe side I have added it.

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak · 2020-07-18T18:41:16Z

The new logic looks good.
A few more nits. We can merge once those are addressed.

Implemented the suggested changes and also updated the flag name.

deepthi · 2020-07-18T20:29:12Z

Very nice work! This has been a long-standing request.

arindamnayak added 11 commits June 14, 2020 00:23

Initial commit for PITR

b1113c7

- Added flags for restore to time - Added flags for binlog server details Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Initial commit for PITR -2

21b7a04

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

PITR changes

28df598

- added ability to fetch the GTID from the restore time - added ability to apply the binlogs till the above GTID Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Added some validation and fixes while testing

ddf1ee9

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

updated vstream to work with go routine and return the gitd back via …

94e6529

…chan Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

added fix for replication

be17562

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Added timeout of gtid to time lookup

0b7a255

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

added last method to gtid

6805b62

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

added wait for gtid catch up

f4b8375

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

added go version of wait till postion

de7c4c0

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Fix the pos.atleast with required gtid

51864ca

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak requested a review from sougou as a code owner July 3, 2020 18:47

arindamnayak linked an issue Jul 3, 2020 that may be closed by this pull request

RFC: Point in time recovery (PITR) Part 2 #6267

Closed

arindamnayak marked this pull request as draft July 3, 2020 18:48

arindamnayak requested a review from deepthi July 3, 2020 18:48

arindamnayak added 3 commits July 4, 2020 16:17

sync with vitess/master

239ebb0

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Added mariadb last gtid impl

ca96cbc

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

fix build issue

d5bc793

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch from e2fefaf to d5bc793 Compare July 5, 2020 11:02

deepthi requested a review from aquarapid July 6, 2020 15:53

added rippled dependency in docker

1a800d9

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch 2 times, most recently from 1bffb9e to dc8af79 Compare July 9, 2020 12:12

added pitr testcase

47bce65

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch 2 times, most recently from fb065be to a989cf2 Compare July 10, 2020 18:29

checked in the binary for testing

a9bbf8e

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch from a989cf2 to a9bbf8e Compare July 10, 2020 18:50

Added rippled as extra bin path

4acba2e

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch from 2aa48ef to 4acba2e Compare July 11, 2020 10:37

arindamnayak added 2 commits July 14, 2020 11:30

merged with latest master

53dded5

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

rearranged the testcases

943903c

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak commented Jul 14, 2020

View reviewed changes

go/vt/vttablet/tabletmanager/restore.go Outdated Show resolved Hide resolved

arindamnayak commented Jul 14, 2020

View reviewed changes

go/vt/vttablet/tabletmanager/restore.go Outdated Show resolved Hide resolved

arindamnayak commented Jul 14, 2020

View reviewed changes

arindamnayak added 5 commits July 15, 2020 23:55

include changes for improving the error messages and comments, variab…

d24e79e

…le names Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

merge docker tests to matrix

653ad37

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

updated workflow

f694c31

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Rervert back to individual action

ff5a0d2

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

remove the skipmetacheck from unneeded place

7bb7863

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi reviewed Jul 16, 2020

View reviewed changes

Implemented suggestion in PR comments

c500654

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

handle the case where upto snapshot time there is no binlog event

eaf2454

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

aquarapid reviewed Jul 17, 2020

View reviewed changes

arindamnayak added 2 commits July 18, 2020 00:03

fix the vstream end of events

6abb509

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

improved the comment and rearrange the tests

ecf278b

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Replace string check with gtid.pos.atleast

5b64ddc

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi approved these changes Jul 18, 2020

View reviewed changes

updated flag name and improvement on code comments

878c871

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi merged commit e86a967 into vitessio:master Jul 18, 2020

arindamnayak deleted the pitr-2 branch July 21, 2020 18:21

deepthi added this to the v7.0 milestone Jul 27, 2020

ameetkotian mentioned this pull request Aug 20, 2020

Slack vitess 2020.08.19.r0 tinyspeck/vitess#180

Merged

shlomi-noach mentioned this pull request Sep 15, 2022

RFC: incremental backup and point-in-time recovery #11227

Closed

shlomi-noach mentioned this pull request Aug 28, 2024

Backup/restore: remove legacy binlog-server based PITR code #16673

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented PITR-2 #6408

Implemented PITR-2 #6408

arindamnayak commented Jul 3, 2020 •

edited by deepthi

Loading

arindamnayak Jul 14, 2020 •

edited

Loading

sougou Jul 16, 2020

deepthi left a comment

deepthi Jul 16, 2020

arindamnayak Jul 16, 2020

deepthi Jul 16, 2020

arindamnayak commented Jul 16, 2020

aquarapid Jul 17, 2020

arindamnayak Jul 18, 2020

deepthi commented Jul 17, 2020

arindamnayak commented Jul 18, 2020

deepthi left a comment •

edited

Loading

deepthi Jul 18, 2020

arindamnayak Jul 18, 2020

arindamnayak commented Jul 18, 2020

deepthi commented Jul 18, 2020

Implemented PITR-2 #6408

Implemented PITR-2 #6408

Conversation

arindamnayak commented Jul 3, 2020 • edited by deepthi Loading

arindamnayak Jul 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deepthi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arindamnayak commented Jul 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deepthi commented Jul 17, 2020

arindamnayak commented Jul 18, 2020

deepthi left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arindamnayak commented Jul 18, 2020

deepthi commented Jul 18, 2020

arindamnayak commented Jul 3, 2020 •

edited by deepthi

Loading

arindamnayak Jul 14, 2020 •

edited

Loading

deepthi left a comment •

edited

Loading