Skip to content

Commit

Permalink
[SPARK-50525][SQL][TESTS][FOLLOWUP] Fix `DataFrameSuite.repartition b…
Browse files Browse the repository at this point in the history
…y MapType` test assumption

### What changes were proposed in this pull request?

This is a follow-up to recover the NON-ANSI mode CI failure by adding a test assumption clearly.
- #49144

### Why are the changes needed?

**BEFORE**
```
$ SPARK_ANSI_SQL_MODE=false build/sbt "sql/testOnly *.DataFrameSuite -- -z MapType"
[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.sql.DataFrameSuite
```

**AFTER**
```
$ SPARK_ANSI_SQL_MODE=false build/sbt "sql/testOnly *.DataFrameSuite -- -z MapType"
[info] All tests passed.
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manually test with `SPARK_ANSI_SQL_MODE=false`.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #49457 from dongjoon-hyun/SPARK-50525.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Jan 13, 2025
1 parent 898f7af commit e284b2c
Showing 1 changed file with 17 additions and 15 deletions.
32 changes: 17 additions & 15 deletions sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
Original file line number Diff line number Diff line change
Expand Up @@ -430,21 +430,23 @@ class DataFrameSuite extends QueryTest

test("repartition by MapType") {
Seq("int", "long", "float", "double", "decimal(10, 2)", "string", "varchar(6)").foreach { dt =>
val df = spark.range(20)
.withColumn("c1",
when(col("id") % 3 === 1, typedLit(Map(1 -> 1)))
.when(col("id") % 3 === 2, typedLit(Map(1 -> 1, 2 -> 2)))
.otherwise(typedLit(Map(2 -> 2, 1 -> 1))).cast(s"map<$dt, $dt>"))
.withColumn("c2", typedLit(Map(1 -> null)).cast(s"map<$dt, $dt>"))
.withColumn("c3", lit(null).cast(s"map<$dt, $dt>"))

assertPartitionNumber(df.repartition(4, col("c1")), 2)
assertPartitionNumber(df.repartition(4, col("c2")), 1)
assertPartitionNumber(df.repartition(4, col("c3")), 1)
assertPartitionNumber(df.repartition(4, col("c1"), col("c2")), 2)
assertPartitionNumber(df.repartition(4, col("c1"), col("c3")), 2)
assertPartitionNumber(df.repartition(4, col("c1"), col("c2"), col("c3")), 2)
assertPartitionNumber(df.repartition(4, col("c2"), col("c3")), 2)
withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
val df = spark.range(20)
.withColumn("c1",
when(col("id") % 3 === 1, typedLit(Map(1 -> 1)))
.when(col("id") % 3 === 2, typedLit(Map(1 -> 1, 2 -> 2)))
.otherwise(typedLit(Map(2 -> 2, 1 -> 1))).cast(s"map<$dt, $dt>"))
.withColumn("c2", typedLit(Map(1 -> null)).cast(s"map<$dt, $dt>"))
.withColumn("c3", lit(null).cast(s"map<$dt, $dt>"))

assertPartitionNumber(df.repartition(4, col("c1")), 2)
assertPartitionNumber(df.repartition(4, col("c2")), 1)
assertPartitionNumber(df.repartition(4, col("c3")), 1)
assertPartitionNumber(df.repartition(4, col("c1"), col("c2")), 2)
assertPartitionNumber(df.repartition(4, col("c1"), col("c3")), 2)
assertPartitionNumber(df.repartition(4, col("c1"), col("c2"), col("c3")), 2)
assertPartitionNumber(df.repartition(4, col("c2"), col("c3")), 2)
}
}
}

Expand Down

0 comments on commit e284b2c

Please sign in to comment.