[SPARK-27893][SQL][PYTHON][FOLLOW-UP] Allow Scalar Pandas and Python UDFs can be tested with Scala test base #24945
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
commented
Jun 24, 2019
| sys.props("spark.test.home") | ||
| assert(sys.props.contains("spark.test.home") || | ||
| sys.env.contains("SPARK_HOME"), "spark.test.home or SPARK_HOME is not set.") | ||
| sys.props.getOrElse("spark.test.home", sys.env("SPARK_HOME")) |
There was a problem hiding this comment.
This is for IDE case. spark.test.home can be missing if we run the tests in IDE without any other settings. In that case, it falls back to SPARK_HOME.
There was a problem hiding this comment.
Should we add a comment for this reason?
There was a problem hiding this comment.
oops, I missed this. Actually there are multiple places like this. Let me fix them together later separately.
|
Test build #106814 has finished for PR 24945 at commit
|
viirya
reviewed
Jun 24, 2019
sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
Outdated
Show resolved
Hide resolved
sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
Outdated
Show resolved
Hide resolved
viirya
reviewed
Jun 24, 2019
| * }}} | ||
| * | ||
| * To use it in Scala API and SQL: | ||
| * {{{ | ||
| * sql("SELECT udf_name(1)") | ||
| * spark.select(expr("udf_name(1)") | ||
| * spark.range(10).select(expr("udf_name(id)") | ||
| * spark.range(10).select(pandasTestUDF($"id")) |
There was a problem hiding this comment.
Will we use it? In SQLQueryTestSuite, I think udfs are all registered for UDFTestCase?
There was a problem hiding this comment.
Ah this one will be used at #24946
|
Test build #106843 has finished for PR 24945 at commit
|
viirya
approved these changes
Jun 25, 2019
|
Thank you @viirya. This is not invasive at all. Let me merge it. Merged to master. |
kiku-jw
pushed a commit
to kiku-jw/spark
that referenced
this pull request
Jun 26, 2019
…UDFs can be tested with Scala test base
## What changes were proposed in this pull request?
After this PR, we can test Pandas and Python UDF as below **in Scala side**:
```scala
import IntegratedUDFTestUtils._
val pandasTestUDF = TestScalarPandasUDF("udf")
spark.range(10).select(pandasTestUDF($"id")).show()
```
## How was this patch tested?
Manually tested.
Closes apache#24945 from HyukjinKwon/SPARK-27893-followup.
Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
|
Late +1, very nice! |
HyukjinKwon
added a commit
to HyukjinKwon/spark
that referenced
this pull request
Aug 15, 2019
…h in EpochTracker (to support Python UDFs) This PR proposes to use `InheritableThreadLocal` instead of `ThreadLocal` for current epoch in `EpochTracker`. Python UDF needs threads to write out to and read it from Python processes and when there are new threads, previously set epoch is lost. After this PR, Python UDFs can be used at Structured Streaming with the continuous mode. The test cases were written on the top of apache#24945. Unit tests were added. Manual tests. Closes apache#24946 from HyukjinKwon/SPARK-27234. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
After this PR, we can test Pandas and Python UDF as below in Scala side:
How was this patch tested?
Manually tested.