Skip to content

Commit

Permalink
[SPARK-17346][SQL][TESTS] Fix the flaky topic deletion in KafkaSource…
Browse files Browse the repository at this point in the history
…StressSuite

## What changes were proposed in this pull request?

A follow up Pr for SPARK-17346 to fix flaky `org.apache.spark.sql.kafka010.KafkaSourceStressSuite`.

Test log: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.4/1855/testReport/junit/org.apache.spark.sql.kafka010/KafkaSourceStressSuite/_It_is_not_a_test_/

Looks like deleting the Kafka internal topic `__consumer_offsets` is flaky. This PR just simply ignores internal topics.

## How was this patch tested?

Existing tests.

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #15384 from zsxwing/SPARK-17346-flaky-test.
  • Loading branch information
zsxwing authored and tdas committed Oct 11, 2016
1 parent c8c0906 commit 75b9e35
Showing 1 changed file with 1 addition and 8 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ import java.util.concurrent.atomic.AtomicInteger
import scala.util.Random

import org.apache.kafka.clients.producer.RecordMetadata
import org.scalatest.BeforeAndAfter
import org.scalatest.time.SpanSugar._

import org.apache.spark.sql.execution.streaming._
Expand Down Expand Up @@ -344,7 +343,7 @@ class KafkaSourceSuite extends KafkaSourceTest {
}


class KafkaSourceStressSuite extends KafkaSourceTest with BeforeAndAfter {
class KafkaSourceStressSuite extends KafkaSourceTest {

import testImplicits._

Expand All @@ -358,12 +357,6 @@ class KafkaSourceStressSuite extends KafkaSourceTest with BeforeAndAfter {
start + Random.nextInt(start + end - 1)
}

after {
for (topic <- testUtils.getAllTopicsAndPartitionSize().toMap.keys) {
testUtils.deleteTopic(topic)
}
}

test("stress test with multiple topics and partitions") {
topics.foreach { topic =>
testUtils.createTopic(topic, partitions = nextInt(1, 6))
Expand Down

0 comments on commit 75b9e35

Please sign in to comment.