Skip to content

Commit

Permalink
[SPARK-16251][SPARK-20200][CORE][TEST] Flaky test: org.apache.spark.r…
Browse files Browse the repository at this point in the history
…dd.LocalCheckpointSuite.missing checkpoint block fails with informative message

## What changes were proposed in this pull request?

Currently we don't wait to confirm the removal of the block from the slave's BlockManager, if the removal takes too much time, we will fail the assertion in this test case.
The failure can be easily reproduced if we sleep for a while before we remove the block in BlockManagerSlaveEndpoint.receiveAndReply().

## How was this patch tested?
N/A

Author: Xingbo Jiang <xingbo.jiang@databricks.com>

Closes #18314 from jiangxb1987/LocalCheckpointSuite.

(cherry picked from commit 7dc3e69)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
jiangxb1987 authored and cloud-fan committed Jun 15, 2017
1 parent 0239b16 commit 7efd475
Showing 1 changed file with 8 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@

package org.apache.spark.rdd

import scala.concurrent.duration._

import org.scalatest.concurrent.Eventually.{eventually, interval, timeout}

import org.apache.spark.{LocalSparkContext, SparkContext, SparkException, SparkFunSuite}
import org.apache.spark.storage.{RDDBlockId, StorageLevel}

Expand Down Expand Up @@ -168,6 +172,10 @@ class LocalCheckpointSuite extends SparkFunSuite with LocalSparkContext {
// Collecting the RDD should now fail with an informative exception
val blockId = RDDBlockId(rdd.id, numPartitions - 1)
bmm.removeBlock(blockId)
// Wait until the block has been removed successfully.
eventually(timeout(1 seconds), interval(100 milliseconds)) {
assert(bmm.getBlockStatus(blockId).isEmpty)
}
try {
rdd.collect()
fail("Collect should have failed if local checkpoint block is removed...")
Expand Down

0 comments on commit 7efd475

Please sign in to comment.