[SPARK-21394][SPARK-21432][PYTHON] Reviving callable object/partial function support in UDF in PySpark

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to avoid __name__ in the tuple naming the attributes assigned directly from the wrapped function to the wrapper function, and use self._name (func.__name__ or obj.__class__.name__).

After SPARK-19161, we happened to break callable objects as UDFs in Python as below:

from pyspark.sql import functions


class F(object):
    def __call__(self, x):
        return x

foo = F()
udf = functions.udf(foo)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../spark/python/pyspark/sql/functions.py", line 2142, in udf
    return _udf(f=f, returnType=returnType)
  File ".../spark/python/pyspark/sql/functions.py", line 2133, in _udf
    return udf_obj._wrapped()
  File ".../spark/python/pyspark/sql/functions.py", line 2090, in _wrapped
    @functools.wraps(self.func)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/functools.py", line 33, in update_wrapper
    setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: F instance has no attribute '__name__'

This worked in Spark 2.1:

from pyspark.sql import functions


class F(object):
    def __call__(self, x):
        return x

foo = F()
udf = functions.udf(foo)
spark.range(1).select(udf("id")).show()

+-----+
|F(id)|
+-----+
|    0|
+-----+

After

from pyspark.sql import functions


class F(object):
    def __call__(self, x):
        return x

foo = F()
udf = functions.udf(foo)
spark.range(1).select(udf("id")).show()

+-----+
|F(id)|
+-----+
|    0|
+-----+

In addition, we also happened to break partial functions as below:

from pyspark.sql import functions
from functools import partial


partial_func = partial(lambda x: x, x=1)
udf = functions.udf(partial_func)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../spark/python/pyspark/sql/functions.py", line 2154, in udf
    return _udf(f=f, returnType=returnType)
  File ".../spark/python/pyspark/sql/functions.py", line 2145, in _udf
    return udf_obj._wrapped()
  File ".../spark/python/pyspark/sql/functions.py", line 2099, in _wrapped
    @functools.wraps(self.func, assigned=assignments)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/functools.py", line 33, in update_wrapper
    setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: 'functools.partial' object has no attribute '__module__'

This worked in Spark 2.1:

from pyspark.sql import functions
from functools import partial


partial_func = partial(lambda x: x, x=1)
udf = functions.udf(partial_func)
spark.range(1).select(udf()).show()

+---------+
|partial()|
+---------+
|        1|
+---------+

After

from pyspark.sql import functions
from functools import partial


partial_func = partial(lambda x: x, x=1)
udf = functions.udf(partial_func)
spark.range(1).select(udf()).show()

+---------+
|partial()|
+---------+
|        1|
+---------+

How was this patch tested?

Unit tests in python/pyspark/sql/tests.py and manual tests.

[HyukjinKwon]

cc @holdenk, could you take a look when you have some time?

[SparkQA]

Test build #79572 has finished for PR 18615 at commit 6d3ef48.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

Thanks for catching this and the quick fix. One suggestion that we should leave a comment about the intention behind what we are doing but otherwise looks reasonable.

[holdenk]

I'd put a comment here saying what this is for just so that when we see this next year we don't forget why we stripped out name from the things we asked functools to assign.

[HyukjinKwon]

Sure, thanks @holdenk. I just address your comments.

[SparkQA]

Test build #79582 has finished for PR 18615 at commit 2317226.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

(gentle ping @holdenk)

[holdenk]

LGTM will merge tomorrow unless anyone else has anything to say.

[viirya]

here For example -> here. For example
Or:
here For example -> here, for example

[HyukjinKwon]

Thanks @viirya.

@holdenk, I happened to find another issue and tried to fix both here together.

[viirya]

LGTM

[SparkQA]

Test build #79653 has finished for PR 18615 at commit 4950b8d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #79654 has finished for PR 18615 at commit 8167ea1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

The update looks good to me. I'll merge this to master.

[HyukjinKwon]

Thank you @holdenk and @viirya.

[holdenk]

Merged into master.