Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

slack-vitess-r14.0.5: several vtorc backports from v15 #118

Closed
wants to merge 15 commits into from
Closed

slack-vitess-r14.0.5: several vtorc backports from v15 #118

wants to merge 15 commits into from

Conversation

timvaillancourt
Copy link
Member

@timvaillancourt timvaillancourt commented Aug 16, 2023 •
edited
Loading

Description

This PR backports several vtorc improvements from the v15 release

Related PR(s)

  1. Delete all legacy sharding related code vitessio/vitess#10278
  2. Remove v2 resharding fields vitessio/vitess#10409
  3. [cmd/*] Switch to pflag for all CLI flag parsing vitessio/vitess#10619
    • This PR impacts all binaries but is necessary for other backports
    • Backporting this will make other backports easier for the DS Defense project where we usually need to bridge this gap manually
  4. VTOrc Cleanup: Remove KV stores vitessio/vitess#10645
  5. Use TMC RPCs in VTOrc vitessio/vitess#10664
  6. BugFix: VTOrc should repair replication if either replication thread is stopped vitessio/vitess#10786
  7. BugFix: Vttablet semi-sync settings on startup vitessio/vitess#10881
  8. Replicas should be able to heal if replication is not initialised properly vitessio/vitess#10943
  9. Getting rid of external logging vitessio/vitess#11085
  10. Moving math package from external libarary vitessio/vitess#11147
  11. Nil-check errors before printing them in VTOrc vitessio/vitess#11156
  12. Prevent martini from logging in VTOrc vitessio/vitess#11173

Checklist

  • "Backport to:" labels have been added if this change should be back-ported
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on the CI
  • Documentation was added or is not required

Deployment Notes

Andrew Mason and others added 5 commits August 16, 2023 16:59
@timvaillancourt
[cmd/*] Switch to pflag for all CLI flag parsing (vitessio#10619)
f964893 
* Switch to `pflag` for all parsing

This transparently swaps the cli parsing library used by `internal/flag`
from the standard library `flag` package to `spf13/pflag`.

It also introduces hook points for packages throughout the vitess codebase
to register their flags for either all commands using `servenv` or a
particular subset of commands. This allows these packages to continue to
define their flag variables in a package-private way, but without
polluting the global flagset.

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* Workaround exit code difference between stdlib `flag` and `pflag`

tl;dr stdlib `flag` has [this][1] and `pflag` does not

[1]: golang/go@dcf0929

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* adjust test data for difference in spacing between pflag/stdflag

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* update lingering legacy flag tests

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* Update vtgate/tabletgateway.go to use new interface to isolate flags, update test data

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* update flags in java test code

Signed-off-by: Andrew Mason <andrew@planetscale.com>
@mattlord @timvaillancourt
Port vtorc CLI to servenv and pflag (vitessio#10911)
e08e3b1 
* Port vtorc CLI to servenv and pflag

This is a mea culpa on my part; we missed this one before the v14 ship
date (because I was unsure the status of vtorc GA and forgot to ask).
Consequently, we have to do some extra work to avoid breaking
compatibility.

Therefore, I introduce a function to inline transform the args passed by
the user on the CLI from those that worked with stdlib flag into those
that work with pflag. If there is a diff, we print a warning to the user
so they know how to update their invocation, so they don't experience
issues when we drop this shim code in v16.

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* fix help text and parsing

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* fixup! wip: port vtorc CLI to servenv and pflag

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* Add documentation to the transformation func

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* fix flag name

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* Remove app.HelpCommand branches, which describe unsupported functionality

Signed-off-by: Andrew Mason <andrew@planetscale.com>

* More cleanup: remove HelpCommand and associated code

Signed-off-by: Andrew Mason <andrew@planetscale.com>

Co-authored-by: Matt Lord <mattalord@gmail.com>
@GuptaManan100 @timvaillancourt
VTOrc Cleanup: Remove KV stores (vitessio#10645)
a7c9793 
* feat: remove kv_store from vtorc codebase

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: add import for side-effects for fixing tests

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 @timvaillancourt
Use TMC RPCs in VTOrc (vitessio#10664)
835f4cd 
* feat: remove redundant semi-sync setting in fixPrimary since UndoDemotePrimary rpc already does that

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: use tmc rpcs instead of vtorc specific code in fixReplica

Signed-off-by: Manan Gupta <manan@planetscale.com>

* refactor: refactor code to be cleaner and return the correct value of recovery attempted

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 @timvaillancourt
BugFix: VTOrc should repair replication if either replication thread …
fd83e41 
…is stopped (vitessio#10786)

* feat: add failing vtorc test

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: consider replication stopped if either thread is stopped

Signed-off-by: Manan Gupta <manan@planetscale.com>
@salesforce-cla
Copy link

Thanks for the contribution! Before we can merge this, we need @ajm188 @GuptaManan100 to sign the Salesforce Inc. Contributor License Agreement.

GuptaManan100 and others added 3 commits August 16, 2023 18:39
@GuptaManan100 @timvaillancourt
feat: fix wrangler tests, since they expect 2 more commands on vttabl…
2c4c169 
…et startup

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 @timvaillancourt
Replicas should be able to heal if replication is not initialised pro…
d0f4614 
…perly (vitessio#10943)

* feat: add code to also reset replication parameters in setReplicationSourceLocked when required

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: fix tests to reflect the change

Signed-off-by: Manan Gupta <manan@planetscale.com>
@rsajwani @timvaillancourt
Getting rid of external logging (vitessio#11085)
9beab39 
* Getting rid of external logging

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fix unit test failures

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fix static code check

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

* Fixing linter error

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
@salesforce-cla
Copy link

Thanks for the contribution! Before we can merge this, we need @ajm188 @GuptaManan100 @rsajwani to sign the Salesforce Inc. Contributor License Agreement.

rsajwani and others added 6 commits August 16, 2023 18:59
@rsajwani @timvaillancourt
Moving math package from external libarary (vitessio#11147)
3a2620a 
Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>
@GuptaManan100 @timvaillancourt
feat: nil-check errors before printing them (vitessio#11156)
f68326f 
Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 @timvaillancourt
Prevent martini from logging in VTOrc (vitessio#11173)
450ab06 
* feat: prevent martini from logging

Signed-off-by: Manan Gupta <manan@planetscale.com>

* test: add test for checking martini does not log

Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>
@timvaillancourt
Fix import
f7eefe2 
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
@arvind-murty @timvaillancourt
Remove v2 resharding fields (vitessio#10409)
373f9e2 
* cleanup: remove sharding_column_name and sharding_column_type

Signed-off-by: Arvind Murty <arvind.murty@gmail.com>

* cleanup: remove sharding_column_name and sharding_column_type

Signed-off-by: Arvind Murty <arvind.murty@gmail.com>

* cleanup: remove sharding_column_name and sharding_column_type

Signed-off-by: Arvind Murty <arvind.murty@gmail.com>

* cleanup: remove sharding_column_name and sharding_column_type

Signed-off-by: Arvind Murty <arvind.murty@gmail.com>

* generate vtadmin files

Signed-off-by: Arvind Murty <arvind.murty@gmail.com>

* cleanup: remove sharding_column_name and sharding_column_type from vtadmin

Signed-off-by: Arvind Murty <arvind.murty@gmail.com>

* Merge from main

Signed-off-by: Arvind Murty <arvind.murty@gmail.com>
@timvaillancourt
go fmt
68cefa7 
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
@salesforce-cla
Copy link

Thanks for the contribution! Before we can merge this, we need @ajm188 @arvind-murty @rsajwani @GuptaManan100 to sign the Salesforce Inc. Contributor License Agreement.

@mattlord @timvaillancourt
Delete all legacy sharding related code (vitessio#10278)
86c2e86 
* Delete all legacy sharding related code

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Move used until functions from initialsharding to cluster

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Remove vtctl commands

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Kill vtworker and SetKeyspaceServedFrom cmd

Signed-off-by: Matt Lord <mattalord@gmail.com>

* WaitForDrain related stragglers

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Legacy local straggler workflow

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Get rid of SetKeyspaceShardingInfo & wait for drain stragglers

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Remove vtworker stragglers

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Update throttlerservice protobuf

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Rename test 24, add hashicorp vault test to it (now mysql_server_vault)

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Remove last mentiones of legacy sharding in vtctl

Signed-off-by: Matt Lord <mattalord@gmail.com>

* remove binlog_use_v3_resharding_mode

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Address review comments

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Address review comments

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Correct vtgate help output

Signed-off-by: Matt Lord <mattalord@gmail.com>
@salesforce-cla
Copy link

Thanks for the contribution! Before we can merge this, we need @mattlord @ajm188 @arvind-murty @rsajwani @GuptaManan100 to sign the Salesforce Inc. Contributor License Agreement.

@github-actions
Copy link

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

  • Push additional commits to the associated branch.
  • Remove the stale label.
  • Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

@github-actions github-actions bot added the Stale label Sep 16, 2023
@github-actions
Copy link

This PR was closed because it has been stale for 7 days with no activity.

@github-actions github-actions bot closed this Sep 24, 2023
@timvaillancourt timvaillancourt deleted the bp-v15-vtorc-slack-vitess-r14.0.5 branch October 9, 2023 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
cla:missing Stale upstream-backport An upstream backport v14 v15-backport
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@timvaillancourt @GuptaManan100 @rsajwani @arvind-murty @mattlord