Skip to content

Commit

Permalink
[SPARK-50251][PYTHON] Add getSystemProperty to PySpark SparkContext
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to add `getSystemProperty` to PySpark `SparkContext` like `setSystemProperty` at Apache Spark 4.0.0.

https://spark.apache.org/docs/4.0.0-preview2/api/python/reference/api/pyspark.SparkContext.setSystemProperty.html

### Why are the changes needed?

Since Apache Spark 0.9.0, `setSystemProperty` has been provided because Python doesn't have JVM's `SystemProperties` concept. This is usefully used like the following.

https://github.com/apache/spark/blob/99d27c9701019a0f534cc13c084895a00badee12/python/pyspark/shell.py#L84

This PR aims to add `getSystemProperty` additionally and symmetrically to provide an easier and better experience for both Spark developers and Spark App developers.

### Does this PR introduce _any_ user-facing change?

No. This is a new API.

### How was this patch tested?

Pass the CIs with newly added doctests.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#48781 from dongjoon-hyun/SPARK-50251.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Nov 7, 2024
1 parent 49b0811 commit d9c596c
Show file tree
Hide file tree
Showing 2 changed files with 23 additions and 0 deletions.
1 change: 1 addition & 0 deletions python/docs/source/reference/pyspark.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ Spark Context APIs
SparkContext.getJobTags
SparkContext.getLocalProperty
SparkContext.getOrCreate
SparkContext.getSystemProperty
SparkContext.hadoopFile
SparkContext.hadoopRDD
SparkContext.listArchives
Expand Down
22 changes: 22 additions & 0 deletions python/pyspark/core/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -554,6 +554,28 @@ def setSystemProperty(cls, key: str, value: str) -> None:
assert SparkContext._jvm is not None
SparkContext._jvm.java.lang.System.setProperty(key, value)

@classmethod
def getSystemProperty(cls, key: str) -> str:
"""
Get a Java system property, such as `java.home`.
.. versionadded:: 4.0.0
Parameters
----------
key : str
The key of a new Java system property.
Examples
--------
>>> sc.getSystemProperty("SPARK_SUBMIT")
'true'
>>> _ = sc.getSystemProperty("java.home")
"""
SparkContext._ensure_initialized()
assert SparkContext._jvm is not None
return SparkContext._jvm.java.lang.System.getProperty(key)

@property
def version(self) -> str:
"""
Expand Down

0 comments on commit d9c596c

Please sign in to comment.