schemadiff: foreign key validation (tables and columns)

[shlomi-noach]

Description

We improve schemadiff's support/validation for foreign keys as follows:

• schemadiff analyzes foreign key dependency across the schema
• it sorts tables by fk dependency (ie parents first, then children)
• it validates that there are no circular dependencies (only circle allowed is a self-referencing table)
• schemadiff: validate columns referenced by FOREIGN KEY #10359 already validates that the local columns covered by a foreign key do in fact exist
• we now also check that the referenced (parent) table exists
• and that referenced columns in the parent table - exist
• and that the foreign key lists same count of columns local-vs-referenced
• and that the column types local-vs-referenced match: InnoDB only allows matching an int to an int, for example, and does not allow int-to-bigint. It does allow different VARCHAR length.

Most of this takes place at the schema level, not at the table level.

Many tests added.

Still todo (in a different PR):

• validate indexes: each foreign key must have an appropriate index that covered columns in local table, and an appropriate index that covers referenced columns in parent table
  • an appropriate index covers the foreign key columns in same order as they appear in the foreign key. It may further extend to cover other columns, but the foreign key columns must be the prefix of the index column list.
• apply(): when adding a foreign key and no appropriate index exists, implicitly add it
• apply(): when creating a new table with foreign key and no appropriate index exists, implicitly add it
• apply(): do not allow drop key if it leaves a foreign key without an appropriate index
• validate uniqueness of constraints name across the schema (foreign keys and check constraints alike)

Related Issue(s)

Foreign keys tracking issue: #11975
schemadiff tracking issue: #10203
Followup to #10359

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.

If a new flag is being introduced:

• Is it really necessary to add this flag?
• Flag names should be clear and intuitive (as far as possible)
• Help text should be descriptive.
• Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow should be required, the maintainer team should be notified.

Bug fixes

• There should be at least one unit or end-to-end test.
• The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

• There should be some code comments as to why things are implemented the way they are.

New/Existing features

• Should be documented, either by modifying the existing documentation or creating new documentation.
• New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• vtctl command output order should be stable and awk-able.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from VTop, if used there.

[shlomi-noach]

Foreign keys tracking issue: #11975

[deepthi]

Name suggestion: ForeignKeyColumnCountMismatchError. I don't know if users ever see this though so it may not matter too much.

[shlomi-noach]

Renamed

[deepthi]

I don't see a unit test for this new error type. 4 new errors, but only 1 new unit test.

[shlomi-noach]

You must have missed them, the tests are as follows:

ForeignKeyDependencyUnresolvedError has these unit tests:

• https://github.com/vitessio/vitess/pull/11944/files#diff-22a7f102b05d7044eda2deaa4242957aaffa51bfe99fb9a3e70dda855c6dc544R358-R378
• https://github.com/vitessio/vitess/pull/11944/files#diff-22a7f102b05d7044eda2deaa4242957aaffa51bfe99fb9a3e70dda855c6dc544R307-R310

InvalidColumnInForeignKeyConstraintError has https://github.com/vitessio/vitess/pull/11944/files#diff-22a7f102b05d7044eda2deaa4242957aaffa51bfe99fb9a3e70dda855c6dc544R295-R298

ForeignKeyColumnCountMismatchError has:

• https://github.com/vitessio/vitess/pull/11944/files#diff-22a7f102b05d7044eda2deaa4242957aaffa51bfe99fb9a3e70dda855c6dc544R299-R306
• https://github.com/vitessio/vitess/pull/11944/files#diff-2de8e8fe0a6b562dc9d4566b3de63c1718f3d3df4a938f9ce428ca22a7ebfc66R1481-R1486

MismatchingForeignKeyColumnTypeError has https://github.com/vitessio/vitess/pull/11944/files#diff-22a7f102b05d7044eda2deaa4242957aaffa51bfe99fb9a3e70dda855c6dc544R319-R342

[shlomi-noach]

@deepthi for another look please 🙏

[deepthi]

Sorry I didn't catch this in the previous review. It will be good to rename this as well - to ForeignKeyColumnTypeMismatchError.

[shlomi-noach]

Renamed!

[shlomi-noach]

/cc @dbussink for review

[dbussink]

Mostly some questions but nothing blocking.

[dbussink]

I'd probably write this slight different like this:

for {
     iterationLevel++ 
     ...
}

Reads simpler to me, but no strong opinion really on this.

The other question though is, would it be possible to craft a schema manually that triggers an infinite loop here? Should we have a safeguard for a max level we support?

[shlomi-noach]

would it be possible to craft a schema manually that triggers an infinite loop

The algorithm addresses infinite loops. It leaves out any infinitely dependent views. There's a unit test for that:

vitess/go/vt/schemadiff/schema_test.go

Lines 126 to 135 in ec7e1a8

	func TestNewSchemaFromQueriesLoop(t *testing.T) {
	// v7 and v8 depend on each other
	queries := append(createQueries,
	"create view v7 as select * from v8, t2",
	"create view v8 as select * from t1, v7",
	)
	_, err := NewSchemaFromQueries(queries)
	assert.Error(t, err)
	assert.EqualError(t, err, (&ViewDependencyUnresolvedError{View: "v7"}).Error())
	}

[shlomi-noach]

just clarifying, that the rewrite would be:

for {
     ...
     iterationLevel++ 
}

[shlomi-noach]

fixed

[dbussink]

as an => has an?

[shlomi-noach]

fixed

[dbussink]

Is having an index a logical strict requirement or more an implementation detail of how MySQL uses it to enforce constraints?

[shlomi-noach]

MySQL requires you to have an index over foreign key columns; a future PR enforces that. If you create a foreign key without creating an index, MySQL implicitly creates one on your behalf. The key must cover all the FK columns as a prefix. See future PR.

[shlomi-noach]

The future PR being #12026