[SPARK-49275][SQL] Fix return type nullness of the xpath expression

[chenhao-db]

What changes were proposed in this pull request?

The xpath expression incorrectly marks its return type as array of non-null strings. However, it can actually return an array containing nulls. This can cause NPE in code generation, such as query select coalesce(xpath(repeat('<a></a>', id), 'a')[0], '') from range(1, 2).

Why are the changes needed?

It avoids potential failures in queries that uses the xpath expression.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

A new unit test. It would fail without the change in the PR.

Was this patch authored or co-authored using generative AI tooling?

No.

[chenhao-db]

@HyukjinKwon @dongjoon-hyun Could you help review? Thanks!

[HyukjinKwon]

I believe this was added for Hive compat. Can we check if the behaviour is the same?

[chenhao-db]

@HyukjinKwon Unfortunately, the behavior is not the same. Running the query in Hive gives a different result:

0: jdbc:hive2://> select xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b');
+------+
| _c0  |
+------+
| []   |
+------+

This is a day-1 issue since the xpath expression was introduced to Spark. The root cause is that Hive drops null array elements (source), while Spark doesn't (source).

I'm not sure about the next step. We could make Spark consistent with Hive, but that would be a breaking change for Spark.

[HyukjinKwon]

What's the result before this change in Spark? If it was throwing an exception before, would better match with Hive's for now.

[HyukjinKwon]

If we can justify the difference behaviour from Hive, that works to me too.

[chenhao-db]

In select xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b');, the Spark results before and after the change are the same: [null,null,null]. The change only marks the return type correctly, but doesn't change the return value.

However, in select coalesce(xpath(repeat('<a></a>', id), 'a')[0], '') from range(1, 2), the query would fail with an NPE in generated code before the change, like:

Caused by: java.lang.NullPointerException
  at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
  at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
...

But will return an empty string after the change.

[HyukjinKwon]

ah gotya so it's an existing behaviour okie gotya

[MaxGekk]

@chenhao-db Do the previous Spark versions suffer from the issue too like branch-3.5?

[chenhao-db]

@MaxGekk Yes, I believe this issue has existed since xpath was introduced.

[MaxGekk]

+1, LGTM. Merging to master/3.5.
Thank you, @chenhao-db and @HyukjinKwon for review.

[MaxGekk]

@chenhao-db The changes cause conflicts in branch-3.5. Please, open a separate PR with backport.