Skip to content

Commit

Permalink
Fix 100 char
Browse files Browse the repository at this point in the history
  • Loading branch information
rxin committed Sep 28, 2017
1 parent 9e9627d commit 1de6165
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 11 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ import org.apache.spark.sql.internal.SQLConf
* Ensures that the [[org.apache.spark.sql.catalyst.plans.physical.Partitioning Partitioning]]
* of input data meets the
* [[org.apache.spark.sql.catalyst.plans.physical.Distribution Distribution]] requirements for
* each operator by inserting [[ShuffleExchangeExec]] Operators where required. Also ensure that the
* input partition ordering requirements are met.
* each operator by inserting [[ShuffleExchangeExec]] Operators where required. Also ensure that
* the input partition ordering requirements are met.
*/
case class EnsureRequirements(conf: SQLConf) extends Rule[SparkPlan] {
private def defaultNumPreShufflePartitions: Int = conf.numShufflePartitions
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,9 @@ import org.apache.spark.sql.execution.{ShuffledRowRDD, SparkPlan}
*
* A coordinator is constructed with three parameters, `numExchanges`,
* `targetPostShuffleInputSize`, and `minNumPostShufflePartitions`.
* - `numExchanges` is used to indicated that how many [[ShuffleExchangeExec]]s that will be registered
* to this coordinator. So, when we start to do any actual work, we have a way to make sure that
* we have got expected number of [[ShuffleExchangeExec]]s.
* - `numExchanges` is used to indicated that how many [[ShuffleExchangeExec]]s that will be
* registered to this coordinator. So, when we start to do any actual work, we have a way to
* make sure that we have got expected number of [[ShuffleExchangeExec]]s.
* - `targetPostShuffleInputSize` is the targeted size of a post-shuffle partition's
* input data size. With this parameter, we can estimate the number of post-shuffle partitions.
* This parameter is configured through
Expand All @@ -61,14 +61,14 @@ import org.apache.spark.sql.execution.{ShuffledRowRDD, SparkPlan}
* post-shuffle partitions and pack multiple pre-shuffle partitions with continuous indices
* to a single post-shuffle partition whenever necessary.
* - Finally, this coordinator will create post-shuffle [[ShuffledRowRDD]]s for all registered
* [[ShuffleExchangeExec]]s. So, when a [[ShuffleExchangeExec]] calls `postShuffleRDD`, this coordinator
* can lookup the corresponding [[RDD]].
* [[ShuffleExchangeExec]]s. So, when a [[ShuffleExchangeExec]] calls `postShuffleRDD`, this
* coordinator can lookup the corresponding [[RDD]].
*
* The strategy used to determine the number of post-shuffle partitions is described as follows.
* To determine the number of post-shuffle partitions, we have a target input size for a
* post-shuffle partition. Once we have size statistics of pre-shuffle partitions from stages
* corresponding to the registered [[ShuffleExchangeExec]]s, we will do a pass of those statistics and
* pack pre-shuffle partitions with continuous indices to a single post-shuffle partition until
* corresponding to the registered [[ShuffleExchangeExec]]s, we will do a pass of those statistics
* and pack pre-shuffle partitions with continuous indices to a single post-shuffle partition until
* adding another pre-shuffle partition would cause the size of a post-shuffle partition to be
* greater than the target size.
*
Expand Down Expand Up @@ -101,8 +101,8 @@ class ExchangeCoordinator(
@volatile private[this] var estimated: Boolean = false

/**
* Registers a [[ShuffleExchangeExec]] operator to this coordinator. This method is only allowed to
* be called in the `doPrepare` method of a [[ShuffleExchangeExec]] operator.
* Registers a [[ShuffleExchangeExec]] operator to this coordinator. This method is only allowed
* to be called in the `doPrepare` method of a [[ShuffleExchangeExec]] operator.
*/
@GuardedBy("this")
def registerExchange(exchange: ShuffleExchangeExec): Unit = synchronized {
Expand Down

0 comments on commit 1de6165

Please sign in to comment.