[SPARK-5484][GraphX] Periodically do checkpoint in Pregel #15125
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -362,12 +362,14 @@ class GraphOps[VD: ClassTag, ED: ClassTag](graph: Graph[VD, ED]) extends Seriali | |
| def pregel[A: ClassTag]( | ||
| initialMsg: A, | ||
| maxIterations: Int = Int.MaxValue, | ||
| checkpointInterval: Int = 25, | ||
|
There was a problem hiding this comment. isn't it a breaking change to add a param into the middle of the list? There was a problem hiding this comment. Good point in both cases. I'm wondering if the periodic pregel checkpointing operation should be controlled by a config value instead. Suppose, for example, we create a config key There was a problem hiding this comment. 25 is the value in my test. I checked this value in LDA/ALS, looks like the default value is 10, change it to 10 ? There was a problem hiding this comment. Agree with @mallman , we don't need change the api interface if we use a config value to controll checkpoint interval. I will udpate the PR soon. There was a problem hiding this comment. About the default value, I think we should set it as a positive value to turn on checkpoint operation by default to avoid stackoverflow exception. To align with other implementations in spark, I would like to set 10 as default value. Please let me know if there is any thoughts about this. |
||
| activeDirection: EdgeDirection = EdgeDirection.Either)( | ||
| vprog: (VertexId, VD, A) => VD, | ||
| sendMsg: EdgeTriplet[VD, ED] => Iterator[(VertexId, A)], | ||
| mergeMsg: (A, A) => A) | ||
| : Graph[VD, ED] = { | ||
| Pregel(graph, initialMsg, maxIterations, activeDirection, | ||
| checkpointInterval)(vprog, sendMsg, mergeMsg) | ||
| } | ||
|
|
||
| /** | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,7 +19,10 @@ package org.apache.spark.graphx | |
|
|
||
| import scala.reflect.ClassTag | ||
|
|
||
| import org.apache.spark.graphx.util.PeriodicGraphCheckpointer | ||
| import org.apache.spark.internal.Logging | ||
| import org.apache.spark.rdd.RDD | ||
| import org.apache.spark.util.PeriodicRDDCheckpointer | ||
|
|
||
| /** | ||
| * Implements a Pregel-like bulk-synchronous message-passing API. | ||
|
|
@@ -113,7 +116,8 @@ object Pregel extends Logging { | |
| (graph: Graph[VD, ED], | ||
| initialMsg: A, | ||
| maxIterations: Int = Int.MaxValue, | ||
| activeDirection: EdgeDirection = EdgeDirection.Either, | ||
| checkpointInterval: Int = 25) | ||
|
|
||
| (vprog: (VertexId, VD, A) => VD, | ||
| sendMsg: EdgeTriplet[VD, ED] => Iterator[(VertexId, A)], | ||
| mergeMsg: (A, A) => A) | ||
|
|
@@ -123,16 +127,25 @@ object Pregel extends Logging { | |
| s" but got ${maxIterations}") | ||
|
|
||
| var g = graph.mapVertices((vid, vdata) => vprog(vid, vdata, initialMsg)).cache() | ||
| val graphCheckpointer = new PeriodicGraphCheckpointer[VD, ED]( | ||
| checkpointInterval, graph.vertices.sparkContext) | ||
| graphCheckpointer.update(g) | ||
|
|
||
| // compute the messages | ||
| var messages = GraphXUtils.mapReduceTriplets(g, sendMsg, mergeMsg).cache() | ||
| val messageCheckpointer = new PeriodicRDDCheckpointer[(VertexId, A)]( | ||
| checkpointInterval, graph.vertices.sparkContext) | ||
| messageCheckpointer.update(messages.asInstanceOf[RDD[(VertexId, A)]]) | ||
|
There was a problem hiding this comment. Do we need to update it before the loop? I think it should be enough to do the update in the loop. There was a problem hiding this comment. I agree. What do you think, @dding3? There was a problem hiding this comment. Actually, for the sake of simplicity and consistency, I'm going to suggest we keep the checkpointer update calls but remove all There was a problem hiding this comment. I think we need cache graph/messages here so they don't need to be computed again in the loop. I agree with you and I will keep the checkpointer update calls and remove all .cache calls. |
||
| var activeMessages = messages.count() | ||
|
|
||
| // Loop | ||
| var prevG: Graph[VD, ED] = null | ||
| var i = 0 | ||
| while (activeMessages > 0 && i < maxIterations) { | ||
| // Receive the messages and update the vertices. | ||
| prevG = g | ||
| g = g.joinVertices(messages)(vprog).cache() | ||
|
There was a problem hiding this comment. checkpointer will do persist. So I think we don't need to call cache here. There was a problem hiding this comment. Actually, this may be more subtle than I thought. I'm going to think through this again. There was a problem hiding this comment. Ok. I agree with your observation here, @viirya. |
||
| graphCheckpointer.update(g) | ||
|
|
||
| val oldMessages = messages | ||
| // Send new messages, skipping edges where neither side received a message. We must cache | ||
|
|
@@ -143,6 +156,7 @@ object Pregel extends Logging { | |
| // The call to count() materializes `messages` and the vertices of `g`. This hides oldMessages | ||
|
There was a problem hiding this comment. should the comment here be updated? |
||
| // (depended on by the vertices of g) and the vertices of prevG (depended on by oldMessages | ||
| // and the vertices of g). | ||
| messageCheckpointer.update(messages.asInstanceOf[RDD[(VertexId, A)]]) | ||
| activeMessages = messages.count() | ||
|
|
||
| logInfo("Pregel finished iteration " + i) | ||
|
|
@@ -155,6 +169,8 @@ object Pregel extends Logging { | |
| i += 1 | ||
| } | ||
| messages.unpersist(blocking = false) | ||
|
There was a problem hiding this comment. nit: Let's do There was a problem hiding this comment. Looks like unpersist is a protected method and we cannot access it from Pregel. Add a new public method to unpersist dataset to work around this. |
||
| graphCheckpointer.deleteAllCheckpoints() | ||
| messageCheckpointer.deleteAllCheckpoints() | ||
|
There was a problem hiding this comment. shouldn't this be inside a finally clause to make sure checkpoint data is cleaned up even in error cases? There was a problem hiding this comment. I think when there is an exception during training, if we keep the checkpoints, there is a chance for user to recover from it. I checked in RandomForest/GBT in spark, looks like they only delete the checkpoints when the training successful finished. |
||
| g | ||
| } // end of apply | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's PeriodicRDDCheckpointer, shouldn't this be in the
org.apache.spark.rdd.utilnamespace?