-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-11974][CORE]Not all the temp dirs had been deleted when the JVM exits #9951
Conversation
What's the problem with the original approach? |
It can not delete all element of shutdownDeletePaths. |
The part I don't get is --- where are we deleting elements of shutdownDeletePaths? |
|
Ah ok - can you update the change to add a line of comment saying we need to materialize the paths to delete because deleteRecursively removes items from shutdownDeletePaths as we are traversing through it? |
//we need to materialize the paths to delete because deleteRecursively removes items from //shutdownDeletePaths as we are traversing through it.
Ok, get it @rxin |
@@ -57,7 +57,9 @@ private[spark] object ShutdownHookManager extends Logging { | |||
// Add a shutdown hook to delete the temp dirs when the JVM exits | |||
addShutdownHook(TEMP_DIR_SHUTDOWN_PRIORITY) { () => | |||
logInfo("Shutdown hook called") | |||
shutdownDeletePaths.foreach { dirPath => | |||
//we need to materialize the paths to delete because deleteRecursively removes items from |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need a space after // - otherwise this will fail the style check
Is it OK? @rxin |
Test build #2108 has finished for PR 9951 at commit
|
I've merged this. Thanks. |
…VM exits deleting the temp dir like that ``` scala> import scala.collection.mutable import scala.collection.mutable scala> val a = mutable.Set(1,2,3,4,7,0,8,98,9) a: scala.collection.mutable.Set[Int] = Set(0, 9, 1, 2, 3, 7, 4, 8, 98) scala> a.foreach(x => {a.remove(x) }) scala> a.foreach(println(_)) 98 ``` You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection Author: Zhongshuai Pei <peizhongshuai@huawei.com> Closes #9951 from DoingDone9/Bug_RemainDir. (cherry picked from commit 6b78157) Signed-off-by: Reynold Xin <rxin@databricks.com>
…VM exits deleting the temp dir like that ``` scala> import scala.collection.mutable import scala.collection.mutable scala> val a = mutable.Set(1,2,3,4,7,0,8,98,9) a: scala.collection.mutable.Set[Int] = Set(0, 9, 1, 2, 3, 7, 4, 8, 98) scala> a.foreach(x => {a.remove(x) }) scala> a.foreach(println(_)) 98 ``` You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection Author: Zhongshuai Pei <peizhongshuai@huawei.com> Closes #9951 from DoingDone9/Bug_RemainDir. (cherry picked from commit 6b78157) Signed-off-by: Reynold Xin <rxin@databricks.com>
…VM exits deleting the temp dir like that ``` scala> import scala.collection.mutable import scala.collection.mutable scala> val a = mutable.Set(1,2,3,4,7,0,8,98,9) a: scala.collection.mutable.Set[Int] = Set(0, 9, 1, 2, 3, 7, 4, 8, 98) scala> a.foreach(x => {a.remove(x) }) scala> a.foreach(println(_)) 98 ``` You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection Author: Zhongshuai Pei <peizhongshuai@huawei.com> Closes #9951 from DoingDone9/Bug_RemainDir. (cherry picked from commit 6b78157) Signed-off-by: Reynold Xin <rxin@databricks.com>
deleting the temp dir like that
You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection