[SPARK-43440][PYTHON][CONNECT] Support registration of an Arrow-optimized Python UDF

[xinrong-meng]

What changes were proposed in this pull request?

The PR proposes to provide support for the registration of an Arrow-optimized Python UDF in both vanilla PySpark and Spark Connect.

Why are the changes needed?

Currently, when users register an Arrow-optimized Python UDF, it will be registered as a pickled Python UDF and thus, executed without Arrow optimization.

We should support Arrow-optimized Python UDFs registration and execute them with Arrow optimization.

Does this PR introduce any user-facing change?

Yes. No API changes, but result differences are expected in some cases.

Previously, a registered Arrow-optimized Python UDF will be executed without Arrow optimization.
Now, it will be executed with Arrow optimization, as shown below.

>>> df = spark.range(2)
>>> df.createOrReplaceTempView("df")
>>> from pyspark.sql.functions import udf
>>> @udf(useArrow=True)
... def f(x):
...     return str(x)
... 

>>> spark.udf.register('str_f', f)
<pyspark.sql.udf.UserDefinedFunction object at 0x7fa1980c16a0>

>>> spark.sql("select str_f(id) from df").explain()  # Executed with Arrow optimization
== Physical Plan ==
*(2) Project [pythonUDF0#32 AS f(id)#30]
+- ArrowEvalPython [f(id#27L)#29], [pythonUDF0#32], 101
   +- *(1) Range (0, 2, step=1, splits=16)

Enabling or disabling Arrow optimization can produce result differences in some cases - we are working on minimizing the result differences though.

How was this patch tested?

Unit test.

SPARK-40307

[xinrong-meng]

That's the magic line :).

[HyukjinKwon]

Merged to master.