Skip to content

Commit

Permalink
[SPARK-51048][CORE] Support stop java spark context with exit code
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Considering there is support to stop spark context with required exit code, This PR aims to use the same to add it to java spark context as well.

### Why are the changes needed?

Currently to use `stop(exitCode)` method for  JavaSparkContext, user has to call `javaSparkContextsparkContext.stop(exitCode)`.
Post this PR user can directly do `javaSparkContextsparkContext.stop(exitCode)` to invoke same.

### Does this PR introduce _any_ user-facing change?

No, it introduces an extra method in JavaSparkContext class

### How was this patch tested?

This was tested in internal/production k8 cluster

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#49746 from prathit06/SPARK-51048.

Authored-by: prathit06 <malik.prathit@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
prathit06 authored and HyukjinKwon committed Feb 8, 2025
1 parent a5e905d commit af92420
Showing 1 changed file with 15 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -552,6 +552,21 @@ class JavaSparkContext(val sc: SparkContext) extends Closeable {
sc.stop()
}

/**
* Shut down the SparkContext with exit code that will passed to scheduler backend.
* In client mode, client side may call `SparkContext.stop()` to clean up but exit with
* code not equal to 0. This behavior cause resource scheduler such as `ApplicationMaster`
* exit with success status but client side exited with failed status. Spark can call
* this method to stop SparkContext and pass client side correct exit code to scheduler backend.
* Then scheduler backend should send the exit code to corresponding resource scheduler
* to keep consistent.
*
* @param exitCode Specified exit code that will passed to scheduler backend in client mode.
*/
def stop(exitCode: Int): Unit = {
sc.stop(exitCode)
}

override def close(): Unit = stop()

/**
Expand Down

0 comments on commit af92420

Please sign in to comment.