Skip to content

Commit

Permalink
Refine docs
Browse files Browse the repository at this point in the history
  • Loading branch information
jinxing committed May 22, 2017
1 parent ca65544 commit 66aa56f
Showing 1 changed file with 4 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,8 @@ private[spark] class CompressedMapStatus(

/**
* A [[MapStatus]] implementation that stores the accurate size of huge blocks, which are larger
* than both spark.shuffle.accurateBlockThreshold. It stores the average size of other non-empty
* blocks, plus a bitmap for tracking which blocks are empty.
* than spark.shuffle.accurateBlockThreshold. It stores the average size of other non-empty blocks,
* plus a bitmap for tracking which blocks are empty.
*
* @param loc location where the task is being executed
* @param numNonEmptyBlocks the number of non-empty blocks
Expand Down Expand Up @@ -211,8 +211,8 @@ private[spark] object HighlyCompressedMapStatus {
val size = uncompressedSizes(i)
if (size > 0) {
numNonEmptyBlocks += 1
// Remove the huge blocks from the calculation for average size and have accurate size for
// smaller blocks.
// Huge blocks are not included in the calculation for average size, thus size for smaller
// blocks is more accurate.
if (size < threshold) {
totalSize += size
} else {
Expand Down

0 comments on commit 66aa56f

Please sign in to comment.