[Spark] Fix dependent constraints/generated columns checker for type widening

[Alexvsalexvsalex]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

The current checker of dependent expressions doesn't validate changes for array and map types. For example, usage of type widening could lead to constraints breaks:

scala> sql("CREATE TABLE table (a array<byte>) USING DELTA")
scala> sql("INSERT INTO table VALUES (array(1, -2, 3))")
scala> sql("SELECT hash(a[1]) FROM table").show()
+-----------+
| hash(a[1])|
+-----------+
|-1160545675|
+-----------+

scala> sql("ALTER TABLE table ADD CONSTRAINT ch1 CHECK (hash(a[1]) = -1160545675)")
scala> sql("ALTER TABLE table SET TBLPROPERTIES('delta.enableTypeWidening' = true)")
scala> sql("ALTER TABLE table CHANGE COLUMN a.element TYPE BIGINT")
scala> sql("SELECT hash(a[1]) FROM table").show()
+----------+
|hash(a[1])|
+----------+
|-981642528|
+----------+

scala> sql("INSERT INTO table VALUES (array(1, -2, 3))")
24/11/15 12:53:23 ERROR Utils: Aborting task
com.databricks.sql.transaction.tahoe.schema.DeltaInvariantViolationException: [DELTA_VIOLATE_CONSTRAINT_WITH_VALUES] CHECK constraint ch1 (hash(a[1]) = -1160545675) violated by row with values:

The proposed algorithm is more strict and regards maps, arrays and structs during constraints/generated columns dependencies.

How was this patch tested?

Added new tests for constraints and generated columns used with type widening feature.

Does this PR introduce any user-facing changes?

Due to strictness of the algorithm new potential dangerous type changes will be prohibited. An exception will be thrown in the example above.
But such changes are called in the schema evolution feature mostly that was introduced recently, so it should not affect many users.

[johanl-db]

Thanks for fixing this!

It should be fairly safe to ship with the fallback path to the old behavior in case anything goes wrong.

[bart-samwel]

LG, I'll merge it.