Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds RPCs to vttablet that vtorc requires #10464

Merged
merged 33 commits into from
Jun 20, 2022

Conversation

GuptaManan100
Copy link
Member

@GuptaManan100 GuptaManan100 commented Jun 8, 2022

Description

This PR adds 2 new Tablet manager client RPCs that VTOrc requires -

  • ResetReplicationParameters
    • This RPC stops the replication (if running) and resets the replication parameters like source host, port, etc from the server. It does not however clear out the executed gtid set or retrieved gtid set. In MySQL equivalent this function does STOP REPLICA; RESET REPLICA ALL
    • This RPC is supposed to be used in fixPrimaryHasPrimary in VTOrc to reset the replication parameters on the primary tablet
  • FullStatus
    • This RPC collects information from the server including replication information, semi-sync settings, server settings among others. The full list of outputs in the full status are -
      • ServerID: Server ID configured :- select @@global.server_id
      • ServerUUID: Server UUID configured/assigned :- select @@global.server_uuid
      • ReplicationStatus: Replication status of the server :- show replica status
      • PrimaryStatus: Binlog position and executed gtid set :- show master status
      • GTIDPurged: purged gtid set :- select @@gtid_purged
      • Version: Server version in the format majorVersion.minorVersion.patchRelease
      • VersionComment: Version comment in the server :- select @@global.version_comment
      • ReadOnly: Whether the server is read only :- show variables like 'read_only'
      • BinLogFormat: format of the binlog of the server :- select @@global.binlog_format
      • BinLogRowImage: the way to store changes to a row in the binlog :- select @@global.binlog_row_image
      • LogBinEnabled: Whether logging in the binlog is enabled :- select @@global.log_bin
      • LogReplicaUpdates: Whether the server will also add the updates acquired via replication to the binlog :- select @@global.log_slave_updates
      • GTIDMode: select @@global.gtid_mode
      • SemiSyncPrimaryEnabled: Whether the primary side semi-sync plugin is enabled :- show global variables like 'rpl_semi_sync_master_enabled'
      • SemiSyncReplicaEnabled: Whether the replica side semi-sync plugin is enabled :- show global variables like 'rpl_semi_sync_slave_enabled'
      • SemiSyncPrimaryStatus: Status of the primary side semi-sync :- show status like 'Rpl_semi_sync_master_status'
      • SemiSyncReplicaStatus: Status of the replica side semi-sync :- show status like 'Rpl_semi_sync_slave_status'
      • SemiSyncPrimaryClients: Number of semi-sync clients of the server :- show status like 'semi_sync_primary_clients'
      • SemiSyncPrimaryTimeout: Amount of time to wait for semi-sync ack before reverting to async replication :- show status like 'rpl_semi_sync_master_timeout
      • SemiSyncWaitForReplicaCount: Number of semi-sync acks required by the server to accept a write :- show status like 'rpl_semi_sync_master_wait_for_slave_count,
    • This RPC is meant to be used in ReadTopologyInstanceBufferable to collect information of the MySQL server without querying it directly

Apart from these 2 RPCs, this PR also augments the ReplicationStatus struct and adds the following new fields to it -

  • RelayLogFilePosition - Position of the relay log of the server.
  • SourceUser - The source user for the replica.
  • SQLDelay - The configured delay that the sql thread is supposed to maintain.
  • AutoPosition - Whether GTID auto-positioning is in use
  • UsingGTID - If the MariaDB server is using GTIDs for replication
  • HasReplicationFilters - If there are any replication filters configured
  • SSLAllowed - If the server allows ssl
  • ReplicationLagUnknown - Set true if the replication lag is unknown and parsing Seconds_Behind_Primary failed
    An existing field FileRelayLogPosition has been renamed to RelayLogSourceBinLogEquivalentPosition since it is a more apt variable name. The previous variable name would have been too confusing after introducing RelayLogFilePosition
    A test that checks that the rpc changes are backward compatible has also been added.

Related Issue(s)

Checklist

  • "Backport me!" label has been added if this change should be backported
  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
…mi-sync settings, binlog_row_image

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Cluster management release notes labels Jun 14, 2022
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
…port it

Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
… stored

Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
…present

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 changed the title Adds RPCs to vttablet, that vtorc requires Adds RPCs to vttablet that vtorc requires Jun 15, 2022
@GuptaManan100 GuptaManan100 marked this pull request as ready for review June 15, 2022 18:05
@github-actions
Copy link
Contributor

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has the correct release notes label. release notes none should only be used for PRs that are so trivial that they need not be included.
  • If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive.
  • If a workflow is added or modified, each items in Jobs should be named in order to mark it as required. If the workflow should be required, the GitHub Admin should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.

@GuptaManan100
Copy link
Member Author

GuptaManan100 commented Jun 15, 2022

#10467 uses these introduced RPCs and verifies that they serve their purpose

Comment on lines 45 to 49
// FileRelayLogPosition stores the position of the source tablets binary log
// upto which the IO thread has read and added to the relay log
FileRelayLogPosition Position
// RelayLogFilePosition stores the position in the relay log file
RelayLogFilePosition Position
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two field names are confusing. It is going to be too easy to make a mistake while making changes in future.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, renaming the field FileRelayLogPosition to RelayLogSourceBinLogEquivalentPosition

Copy link
Member

@deepthi deepthi Jun 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😆
Guess that's the best we can do to avoid confusion.
I don't know whether binlog is one word - so Binlog vs BinLog
goes to grep vitessio/vitess
OK, we use Binlog almost everywhere except in a few test files. I know this is super-nitpicky, but could you please change it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay 👍

…ayLogSourceBinLogEquivalentPosition and augment test to make sure rpc changes are backward compatible

Signed-off-by: Manan Gupta <manan@planetscale.com>
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 requested a review from deepthi June 17, 2022 12:57
Copy link
Member

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit, the rest LGTM.
You may merge once that is addressed.

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 merged commit 611a260 into vitessio:main Jun 20, 2022
@GuptaManan100 GuptaManan100 deleted the vtorc-rpcs branch June 20, 2022 05:23
GuptaManan100 added a commit to planetscale/vitess that referenced this pull request Jun 20, 2022
* feat: add vttablet rpc to reset replication parameters

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: added end to end testing for the rpc and fixed bug

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: fix typing error

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add basic full status rpc functionality and add test for it

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add all the fields needed in full status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: moved the test to reparent tests and improved it

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: bug fix for no replication status and no primary status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add version to the full status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add binlog information to full status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: fix the comment explaining the binlog information

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add semi-sync statuses to full status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: call the correct command

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add server uuid and id to full status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: make server_id a uint32 to accept the correct range of values

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add few more fields to the full status like version comment, semi-sync settings, binlog_row_image

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: generate vtadmin proto files

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: add assertion to check binlog row format is read correctly

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: split GTID mode in its own function because mariadb doesn't support it

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: fix parsing of empty mariadb gtid set

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add doucmentation for existing fields in ReplicationStatus

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add relay log file position to the replication status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: augmented full status test to check all the different positions stored

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add additional fields to replication status and read source user

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: read sql delay from show replica status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: read ssl allowed from show replica status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: read has replication filters from show replica status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: read auto position and using gtid from show replica status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add replication lag unknown too to replication status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: return nils from replication and primary postiion if it is not present

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: rename FileRelayLogPosition in replication status output to RelayLogSourceBinLogEquivalentPosition and augment test to make sure rpc changes are backward compatible

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: update vtadmin proto files

Signed-off-by: Manan Gupta <manan@planetscale.com>

* refactor: rename BinLog to binlog in renamed proto field

Signed-off-by: Manan Gupta <manan@planetscale.com>
GuptaManan100 added a commit that referenced this pull request Jun 22, 2022
…10464) (#10546)

* Adds RPCs to vttablet that vtorc requires (#10464)

* feat: add vttablet rpc to reset replication parameters

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: added end to end testing for the rpc and fixed bug

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: fix typing error

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add basic full status rpc functionality and add test for it

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add all the fields needed in full status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: moved the test to reparent tests and improved it

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: bug fix for no replication status and no primary status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add version to the full status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add binlog information to full status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: fix the comment explaining the binlog information

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add semi-sync statuses to full status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: call the correct command

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add server uuid and id to full status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: make server_id a uint32 to accept the correct range of values

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add few more fields to the full status like version comment, semi-sync settings, binlog_row_image

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: generate vtadmin proto files

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: add assertion to check binlog row format is read correctly

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: split GTID mode in its own function because mariadb doesn't support it

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: fix parsing of empty mariadb gtid set

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add doucmentation for existing fields in ReplicationStatus

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add relay log file position to the replication status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: augmented full status test to check all the different positions stored

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add additional fields to replication status and read source user

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: read sql delay from show replica status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: read ssl allowed from show replica status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: read has replication filters from show replica status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: read auto position and using gtid from show replica status output

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: add replication lag unknown too to replication status

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: return nils from replication and primary postiion if it is not present

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: rename FileRelayLogPosition in replication status output to RelayLogSourceBinLogEquivalentPosition and augment test to make sure rpc changes are backward compatible

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: update vtadmin proto files

Signed-off-by: Manan Gupta <manan@planetscale.com>

* refactor: rename BinLog to binlog in renamed proto field

Signed-off-by: Manan Gupta <manan@planetscale.com>

* lint: add a new line to proto file

Signed-off-by: Manan Gupta <manan@planetscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Cluster management Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants