SPARK-3770: Make userFeatures accessible from python

[mdagost]

https://issues.apache.org/jira/browse/SPARK-3770

We need access to the underlying latent user features from python. However, the userFeatures RDD from the MatrixFactorizationModel isn't accessible from the python bindings. I've added a method to the underlying scala class to turn the RDD[(Int, Array[Double])] to an RDD[String]. This is then accessed from the python recommendation.py

[AmplabJenkins]

Can one of the admins verify this patch?

[mengxr]

@mdagost If you convert (Int, Array[Double]) to a java.util.List<Object> (id the first and features the second (without converting to string)), you should be able to get the data correctly on the Python side. If that works, could you add productFeatures as well? Thanks!

@davies

[davies]

@mdagost @mengxr We use Pyrolite to convert Java objects into Python objects, you can get the type mapping here: https://github.com/irmen/Pyrolite

So if we convert (Int, Array[Double]) into Array[Object] or 'JList[Object]', we can get a RDD of tuple(int, array(double)) or list(int, array(double)) in Python

[mdagost]

I'm totally new to Spark, so sorry if these are all dumb questions.

Are you suggesting that I convert the userFeatures RDD[(Int, Array[Double])] to RDD[Array[Object]] ? If so, do you want a helper function for doing that like I did for the string helper, or should I convert the main userFeatures to be of that type?

Also, I'm sure this is dumb, but what exact type of Object are we talking about?

[davies]

We still need this wrapper, but RDD[Array[Object]] is only used for Python API, so it's better to put it in PythonMLLibAPI, maybe more general, like fromTupleRDD, which will convert any RDD[Tuple[,]] into RDD[Array[Any]], Any is similar to Java Object.

[MLnick]

Ideally we would want to expose the actual RDD[(Int, Array)] on the PySpark side, in case they are really large - they can then be collected if need be.

Can we make use the existing pairRDDToPython function to do the conversion?

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala#L120

[jkbradley]

@MLnick @mdagost There are a few functions available which you could use for the serialization, but PythonRDD.javaToPython might be a good option. You can see example usage in recommendation.py

[mdagost]

I've been having trouble getting either PythonRDD.javaToPython or pairRDDToPython to work. But porting the general function I wrote from MatrixFactorizationModel.scala to PythonMLLibAPI is also giving me some trouble. I'll get back to it later this week and try to make some progress...

[AmplabJenkins]

Can one of the admins verify this patch?

[mdagost]

@MLnick It doesn't look like pairRDDToPython does the trick. I tried

def userFeatures(self):
    juf = self._java_model.userFeatures()                                                                                                                                                
    juf = sc._jvm.SerDeUtil.pairRDDToPython(juf, 1)
    return juf

but what comes out when I try to print the result of taking the first element of the RDD is just "[[B@176fa1a5" rather than any kind of nicely formatted python object.

[mdagost]

@davies Your idea of adding something like fromTupleRDD to PythonMLLibAPI seems to be the way to go. I'm just doing some cleanup and will push userFeatures and productFeatures in just a bit.

[mengxr]

@mdagost Thanks for working on the SerDe! I tested it locally and it works correctly, but the unit tests for the added methods are missing. Do you mind adding them? You can follow

https://github.com/mdagost/spark/blob/mf_user_features/python/pyspark/mllib/recommendation.py#L55

Basically, we want to verify that userFeatures/productFeatures returns an RDD of key-value pairs with the correct number of records and for each records the feature dimension is correct.

[mdagost]

Whoops. Forgot the tests :) I'll work on those today.

[mdagost]

@mengxr Unit tests are added. I get some unrelated test failures on my local (everything in recommendation.py, including the new stuff, passes.)

[mengxr]

this is ok to test

[mengxr]

test this please

[SparkQA]

QA tests have started for PR 2636 at commit c98f9e2.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2636 at commit c98f9e2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21994/
Test PASSed.