Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for MariaDB compressed binlog events #786

Merged
merged 2 commits into from
May 6, 2023

Conversation

monder
Copy link
Contributor

@monder monder commented May 4, 2023

@monder monder force-pushed the mariadb-compressed branch from e187ab6 to dd14348 Compare May 4, 2023 11:33
Copy link
Collaborator

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm, did you test these new events?

case MARIADB_UPDATE_ROWS_COMPRESSED_EVENT_V1:
return "MariadbUpdateRowsCompressedEventV1"
case MARIADB_DELETE_ROWS_COMPRESSED_EVENT_V1:
return "MariadbDeleteRowsCompressedEventV1"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can also support MARIADB_START_ENCRYPTION_EVENT and MARIADB_QUERY_COMPRESSED_EVENT

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added MARIADB_QUERY_COMPRESSED_EVENT.
Moved the uncompression functions to mysql package as it is shared.
Screenshot 2023-05-05 at 20 56 58
Screenshot 2023-05-05 at 20 59 41

As for MARIADB_START_ENCRYPTION_EVENT - I added this event only to fill enum values. It is tricky to support it. I added the text description, but since everything (including next event pos) is encrypted this library crashes when it is enabled.
Screenshot 2023-05-05 at 21 21 57

@monder
Copy link
Contributor Author

monder commented May 5, 2023

rest lgtm, did you test these new events?

Yes.

package main

import (
	"context"
	"os"

	"github.com/go-mysql-org/go-mysql/mysql"
	"github.com/go-mysql-org/go-mysql/replication"
)

func main() {
	cfg := replication.BinlogSyncerConfig{
		ServerID: 100,
		Flavor:   "mariadb",
		Host:     "127.0.0.1",
		Port:     3306,
		User:     "root",
		Password: "teletaips",
	}
	syncer := replication.NewBinlogSyncer(cfg)

	gtidSet, _ := mysql.ParseMariadbGTIDSet("")
	streamer, _ := syncer.StartSyncGTID(gtidSet)

	for {
		ev, _ := streamer.GetEvent(context.Background())
		ev.Dump(os.Stdout)
	}
}

When

log_bin_compress=1
log_bin_compress_min_len=10

With binlog-format = row:

=== MariadbUpdateRowsCompressedEventV1 ===
Date: 2023-05-05 20:59:31
Log position: 596
Event size: 69
TableID: 18
Flags: 1
Column count: 2
Values:
--
0:"kk"
1:"jkhkjhhjkhkh"
--
0:"kk"
1:"jkhkjhhjkhokh"

With binlog-format = statement:

=== MariadbQueryCompressedEvent ===
Date: 2023-05-05 20:55:26
Log position: 2183
Event size: 135
Slave proxy ID: 6
Execution time: 0
Error code: 0
Schema: infogram
Query: update store s set s.value = "jkhkjhhjkhkh" where s.key = "kk"
GTIDSet: 0-1-75

Copy link
Collaborator

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@lance6716 lance6716 merged commit df895b5 into go-mysql-org:master May 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot get binlog info from MariaDB 10.2.x when string length greater than 252
2 participants