[SPARK-50251][PYTHON] Add getSystemProperty to PySpark SparkContext
#48781
Conversation
|
Could you review this, @xinrong-meng and @zhengruifeng ? |
c693bf8 to
cdf5ede
Compare
|
Could you review this PR too, @HyukjinKwon ? |
| """ | ||
| SparkContext._ensure_initialized() | ||
| assert SparkContext._jvm is not None | ||
| return SparkContext._jvm.java.lang.System.getProperty(key) |
There was a problem hiding this comment.
three questions:
1, shall we name it to be more specific to the Java?
2, do we need to also introduce it in Scala side?
3, shall we make it a SparkSession method? so that we can implement it in SparkConnect later
also cc @cloud-fan for this new feature.
There was a problem hiding this comment.
To @zhengruifeng , as I wrote in the PR description, this is a symmetric version of the existing setSystemProperty. So, no for (1) and (3). For (2), Scala can access System.getProperty directly already.
|
Could you review this PR, @viirya ? |
|
Thank you so much, @yaooqinn ! |
|
Thank you, @viirya ! |
|
Merged to master. Thank you all. |
|
Late LGTM, thanks! |
|
Thank you, @zhengruifeng |
What changes were proposed in this pull request?
This PR aims to add
getSystemPropertyto PySparkSparkContextlikesetSystemPropertyat Apache Spark 4.0.0.https://spark.apache.org/docs/4.0.0-preview2/api/python/reference/api/pyspark.SparkContext.setSystemProperty.html
Why are the changes needed?
Since Apache Spark 0.9.0,
setSystemPropertyhas been provided because Python doesn't have JVM'sSystemPropertiesconcept. This is usefully used like the following.spark/python/pyspark/shell.py
Line 84 in 99d27c9
This PR aims to add
getSystemPropertyadditionally and symmetrically to provide an easier and better experience for both Spark developers and Spark App developers.Does this PR introduce any user-facing change?
No. This is a new API.
How was this patch tested?
Pass the CIs with newly added doctests.
Was this patch authored or co-authored using generative AI tooling?
No.