Skip to content

Commit

Permalink
Fix orc_write_test.py for Databricks 14.3
Browse files Browse the repository at this point in the history
Fixes NVIDIA#11529.

This commit fixes the failure of
`orc_write_test.py::test_orc_do_not_lowercase_columns` as described
in NVIDIA#11529.

The failure was the result of a change in the exception text in
Databricks 14.3. (It differs from Spark 3.5, on which it is based, and
more closely resembles Spark 4.0.)

Signed-off-by: MithunR <mithunr@nvidia.com>
  • Loading branch information
mythrocks committed Oct 25, 2024
1 parent e31a710 commit b502418
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions integration_tests/src/main/python/orc_write_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -360,8 +360,8 @@ def test_orc_do_not_lowercase_columns(spark_tmp_path):

# The wording of the `is not exists` error message in Spark 4.x is unfortunate, but accurate:
# https://github.com/apache/spark/blob/4501285a49e4c0429c9cf2c105f044e1c8a93d21/python/pyspark/errors/error-conditions.json#L487
expected_error_message = "No StructField named acol" if is_before_spark_400() else \
"Key `acol` is not exists."
expected_error_message = "Key `acol` is not exists." if is_spark_400_or_later() or is_databricks_version_or_later(14, 3) \
else "No StructField named acol"
assert_gpu_and_cpu_writes_are_equal_collect(
# column is uppercase
lambda spark, path: spark.range(0, 1000).select(col("id").alias("Acol")).write.orc(path),
Expand Down

0 comments on commit b502418

Please sign in to comment.