[SPARK-29800][SQL] Rewrite non-correlated EXISTS subquery use ScalaSubquery to optimize perf #26437

AngersZhuuuu · 2019-11-08T10:34:44Z

What changes were proposed in this pull request?

Current catalyst rewrite non-correlated exists subquery to BroadcastNestLoopJoin, it's performance is not good , now we rewrite non-correlated EXISTS subquery to ScalaSubquery to optimize the performance.
We rewrite

 WHERE EXISTS (SELECT A FROM TABLE B WHERE COL1 > 10)

to

 WHERE (SELECT 1 FROM (SELECT A FROM TABLE B WHERE COL1 > 10) LIMIT 1) IS NOT NULL

to avoid build join to solve EXISTS expression.

Why are the changes needed?

Optimize EXISTS performance.

Does this PR introduce any user-facing change?

NO

How was this patch tested?

Manuel Tested

AngersZhuuuu · 2019-11-08T10:37:04Z

@cloud-fan
Use this change in #26431 make result right as PostgresSQL
But I don't know Standard writing and I can't find UT about PlanSubqueries . Can you give some advise?

cloud-fan · 2019-11-08T13:52:07Z

sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala

+
+
+  def updateResult(): Unit = {
+    val rows = plan.executeCollect()


The reason why we don't have a physical plan for Exists is: it's not robust. Collecting the entire result of a query plan at the driver side is very likely to hit OOM. That's why we have to convert Exists to a join.

The reason why we don't have a physical plan for Exists is: it's not robust. Collecting the entire result of a query plan at the driver side is very likely to hit OOM. That's why we have to convert Exists to a join.

We can make it just return rdd.isEmpy() since exists just need to judge if result is empty.

sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala

cloud-fan · 2019-11-08T14:40:34Z

sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala

+
+
+  def updateResult(): Unit = {
+    result = !plan.execute().isEmpty()


seems like this is better to execute a non-correlated EXISTS subquery. Maybe we should update RewritePredicateSubquery to only handle correlated EXISTS subquery. @dilipbiswal what do you think?

seems like this is better to execute a non-correlated EXISTS subquery. Maybe we should update RewritePredicateSubquery to only handle correlated EXISTS subquery. @dilipbiswal what do you think?

Yeah, wait for his advise.

@cloud-fan @AngersZhuuuu Thanks for pinging me. Just for me to understand, since we refer to another pr in this pr.

So we are considering planning the Subqueries appearing inside ON clause as a Join, right ?

Assuming above, so if the query was :

SELECT * FROM T1 JOIN T2 ON T1.C1 = T2.C1 AND T1.C1 EXISTS (SELECT 1 FROM T3 WHERE T1.C1 = T3.C1)

We are considering to plan it as :

(T1 LeftSemi T3 ON T1.C1 = T3.C1) Join T2 ON T1.C1 = T2.C2

This Looks okay to me for inner joins. I am just not sure about outer joins.. What do you think Wenchen ?

Now, coming to the non-correlated subqueries, if we keep it as a PlanExpression and execute it, one thing we have to see is "what is the join strategy thats being picked". Its always going to be broadcast nested loop as it won't be a "equi-join" ? right ?

@dilipbiswal

SELECT * FROM T1 JOIN T2 ON T1.C1 = T2.C1 AND T1.C1 EXISTS (SELECT 1 FROM T3 WHERE T1.C1 = T3.C1)

Is not correct .

You mean below ?

SELECT * FROM T1 JOIN T2 ON T1.C1 = T2.C1 AND EXISTS (SELECT 1 FROM T3 WHERE T1.C1 = T3.C1)

For this type sql we need to change RewritePredicateSubquery as cloud-fan said.

@AngersZhuuuu You r right. Sorry.. i had written it as IN initially and forgot to adjust to exists :-)

Yeah, we need to change RewritePredicateSubquery which handles correlated subquery rewrites. The only thing i am not sure is about the outer joins.

Yeah, we need to change RewritePredicateSubquery which handles correlated subquery rewrites. The only thing i am not sure is about the outer joins.

Yes, outer join is complex, if we do this, we need to add end to end test case cover each case to make sure the final plans are as expected.

cloud-fan · 2019-11-08T14:43:05Z

sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala

+
+  override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+    prepareResult()
+    ExistsSubquery(child, subQuery, result).doGenCode(ctx, ev)


why we create ExistsSubquery to only do codegen? can we put the codegen logic in ExistsExec?

why we create ExistsSubquery to only do codegen? can we put the codegen logic in ExistsExec?

There are conflicts between ExecSubqueryExpression and UnaryExpression，
They are both abstract class.

We don't have to extend UnaryExpression and we can still implement codegen, right?

We don't have to extend UnaryExpression and we can still implement codegen, right?

Done

AngersZhuuuu · 2019-11-10T09:25:09Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala

+            buildJoin(outerPlan, sub, LeftAnti, joinCond)
+          } else {
+            Filter(Not(exists), newFilter)
+          }


@cloud-fan @dilipbiswal
Change here to support non-correct exists subquery run like mentioned in this comment https://github.com/apache/spark/pull/26437/files#r344203937

@AngersZhuuuu I discussed this with Wenchen briefly. Do you think we can safely inject a "LIMIT 1" into our subplan to expedite its execution ? Pl. lets us know what you think ?

@AngersZhuuuu I discussed this with Wenchen briefly. Do you think we can safely inject a "LIMIT 1" into our subplan to expedite its execution ? Pl. lets us know what you think ?

I am also thinking about reduce the execution cost of this sub query.
LIMIT 1 is ok .
My direction is making this execution like Spark Thrift Server's incremental collect.
Only execute one partition.

Discuss these two ways safety and cost?

AngersZhuuuu · 2019-11-10T09:30:11Z

@cloud-fan @dilipbiswal @maropu
For support correlated subquery in join's on condition like #25854 (comment)

We need to update PushPredicateThroughJoin and RewritePredicateSubquery together.
Since in PushPredicateThroughJoin it will transform a subquery condition rewrite as a Filter's condition. This change will be catch by RewritePredicateSubquery, and after RewritePredicateSubquery's transform, the total logical is broken.
I prefer to fix all this after this pr and #26431 finished

cloud-fan · 2019-11-11T07:03:37Z

some more thoughts: I think we can rewrite non-correlated EXISTS subquery to a non-correlated scalar subquery.

e.g. SELECT.. FROM t1 WHERE (SELECT ...) can be converted to SELECT ... FROM t1 WHERE (SELECT count(*) FROM (SELECT ...)) = 0. Then we don't need to create a new expression.

AngersZhuuuu · 2019-11-11T07:22:08Z

some more thoughts: I think we can rewrite non-correlated EXISTS subquery to a non-correlated scalar subquery.

e.g. SELECT.. FROM t1 WHERE (SELECT ...) can be converted to SELECT ... FROM t1 WHERE (SELECT count(*) FROM (SELECT ...)) = 0. Then we don't need to create a new expression.

It's ok to do like this, but it will add one more shuffle action for count(). If result is huge, it's a big cost.

cloud-fan · 2019-11-11T07:43:14Z

If result is huge, it's a big cost.

Yea true, we can add a LIMIT 1 to the scalar subquery before count, then the result won't be huge. This is also how we implement Dataset.isEmpty.

AngersZhuuuu · 2019-11-11T07:50:12Z

If result is huge, it's a big cost.

Yea true, we can add a LIMIT 1 to the scalar subquery before count, then the result won't be huge. This is also how we implement Dataset.isEmpty.

I am not sure if LIMIT 1 will get result with only executing one partition.
My thinking direction. #26437 (comment)

AngersZhuuuu · 2019-11-11T08:04:26Z

@dilipbiswal @cloud-fan With LIMIT 1. the explain result.

 sql(
        """
          | SELECT s1.id FROM s1
          | JOIN s2 ON s1.id = s2.id
          | WHERE EXISTS (SELECT * from s3)
        """.stripMargin).explain(true)


== Parsed Logical Plan ==
'Project ['s1.id]
+- 'Filter exists#258 []
   :  +- 'Project [*]
   :     +- 'UnresolvedRelation [s3]
   +- 'Join Inner, ('s1.id = 's2.id)
      :- 'UnresolvedRelation [s1]
      +- 'UnresolvedRelation [s2]

== Analyzed Logical Plan ==
id: int
Project [id#244]
+- Filter exists#258 []
   :  +- Project [id#256]
   :     +- SubqueryAlias `s3`
   :        +- Project [value#253 AS id#256]
   :           +- LocalRelation [value#253]
   +- Join Inner, (id#244 = id#250)
      :- SubqueryAlias `s1`
      :  +- Project [value#241 AS id#244]
      :     +- LocalRelation [value#241]
      +- SubqueryAlias `s2`
         +- Project [value#247 AS id#250]
            +- LocalRelation [value#247]

== Optimized Logical Plan ==
Project [id#244]
+- Join Inner, (id#244 = id#250)
   :- Project [value#241 AS id#244]
   :  +- Filter exists#258 []
   :     :  +- Project [value#253 AS id#256]
   :     :     +- LocalRelation [value#253]
   :     +- LocalRelation [value#241]
   +- Project [value#247 AS id#250]
      +- LocalRelation [value#247]

== Physical Plan ==
*(2) Project [id#244]
+- *(2) BroadcastHashJoin [id#244], [id#250], Inner, BuildRight
   :- *(2) Project [value#241 AS id#244]
   :  +- *(2) Filter EXISTS subquery#258
   :     :  +- Subquery subquery#258, [id=#126]
   :     :     +- CollectLimit 1
   :     :        +- *(1) Project [value#253 AS id#256]
   :     :           +- *(1) LocalTableScan [value#253]
   :     +- *(2) LocalTableScan [value#241]
   +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint))), [id=#135]
      +- *(1) Project [value#247 AS id#250]
         +- *(1) LocalTableScan [value#247]

cloud-fan · 2019-11-11T08:11:14Z

get result with only executing one partition.

do you mean something like RDD.take? e.g. we can execute the first partition, if it's non-empty, return true, otherwise, execute more partitions

AngersZhuuuu · 2019-11-11T08:20:07Z

do you mean something like RDD.take? e.g. we can execute the first partition, if it's non-empty, return true, otherwise, execute more partitions

Yea, what I want.
Since for LIMIT 1, plan as CollectLimitExec, it will execute all partition then collect local limit ,then collect global limit.

 execute the first partition, if it's non-empty, return true, otherwise, execute more partitions

In this way, we can reduce the subquery's minimize expenses.

  def updateResult(): Unit = {
    result = plan.executeTake(1) == 1
    resultBroadcast = plan.sqlContext.sparkContext.broadcast[Boolean](result)
  }

This way is ok?

cloud-fan · 2019-11-11T08:34:42Z

makes sense, let's keep the new EXISTS expression. Can we add a Project(Nil, ...) on the top to do column pruning? this should be critical for performance.

AngersZhuuuu · 2019-11-11T08:39:35Z

makes sense, let's keep the new EXISTS expression. Can we add a Project(Nil, ...) on the top to do column pruning? this should be critical for performance.

Good suggestion, updated. One point I don't support LIMIT 1 is that it will change final PhysicalPlan's out put. This may make user confused if user don't know clear about this change.

@dilipbiswal For current way, what do you think?

cloud-fan · 2019-11-11T10:31:31Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala

-        case (p, Not(Exists(sub, conditions, _))) =>
-          val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
-          buildJoin(outerPlan, sub, LeftAnti, joinCond)
+        case (p, exists @ Exists(sub, conditions, _)) =>


can we change the beginning instead?

val (withSubquery, withoutSubquery) = splitConjunctivePredicates(condition).partition { cond => SubqueryExpression.hasInOrExistsSubquery(cond) && !isNonCorrelatedExists }

or we change hasInOrExistsSubquery to hasInOrCorrelatedExistsSubquery

hasInOrCorrelatedExistsSubquery

Done

cloud-fan · 2019-11-11T10:33:27Z

sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala

+ * The physical node of exists-subquery. This is for support use exists in join's on condition,
+ * since some join type we can't pushdown exists condition, we plan it here
+ */
+case class ExistsExec(child: Expression,


nit:

case class A( para1: T, para2: T): R ...

nit:

case class A( para1: T, para2: T): R ...

Done

cloud-fan · 2019-11-11T10:38:49Z

sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala

+                      subQuery: String,
+                      plan: BaseSubqueryExec,
+                      exprId: ExprId,
+                      private var resultBroadcast: Broadcast[Boolean] = null)


why we need broadcast? Can we follow the physical scalar subquery?

why we need broadcast? Can we follow the physical scalar subquery?

Make it can be used by each partition, reduce return data size during compute or return result

SparkQA · 2020-01-05T06:21:01Z

Test build #4985 has finished for PR 26437 at commit 8c6060a.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

viirya · 2020-01-05T06:43:21Z

Is the second SQL query wrong (COL1 > 1 -> COL1 > 10) in the PR description?

AngersZhuuuu · 2020-01-05T12:46:39Z

Is the second SQL query wrong (COL1 > 1 -> COL1 > 10) in the PR description?

Write comment error in added rule, thanks for you reminding， could you help to trigger retest

viirya · 2020-01-05T17:52:02Z

retest this please.

viirya · 2020-01-05T18:01:29Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala

@@ -56,7 +56,7 @@ object ReplaceExpressions extends Rule[LogicalPlan] {
 * Rewrite non correlated exists subquery to use ScalarSubquery
 *   WHERE EXISTS (SELECT A FROM TABLE B WHERE COL1 > 10)
 * will be rewrite to


nit: rewritten

nit: rewritten

Got the point, changed...

viirya · 2020-01-05T18:05:12Z

sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala

+    plan.expressions.map(_.collect {
+      case sub: ExecSubqueryExpression => getNumInMemoryTablesRecursively(sub.plan)
+    }.sum).sum


nit:

plan.expressions.flatMap(_.collect { case sub: ExecSubqueryExpression => getNumInMemoryTablesRecursively(sub.plan) }).sum

viirya · 2020-01-05T18:06:14Z

sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala

            |WHERE
-            |NOT EXISTS (SELECT * FROM t1)
+            |NOT EXISTS (SELECT * FROM t1 b WHERE a.i = b.i)


Why need to change the existing test?

Why need to change the existing test?

#26437 (comment)

viirya · 2020-01-05T18:07:52Z

sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala

+      case inMemoryTable @ InMemoryTableScanExec(_, _, relation) =>
+        getNumInMemoryTablesRecursively(relation.cachedPlan) +
+          getNumInMemoryTablesInSubquery(inMemoryTable) + 1
+      case p =>
+        getNumInMemoryTablesInSubquery(p)


Is this change needed for this PR? Looks like not directly related?

Is this change needed for this PR? Looks like not directly related?

#26437 (comment)

viirya · 2020-01-05T18:08:19Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala

+        splitConjunctivePredicates(condition)
+          .partition(SubqueryExpression.hasInOrCorrelatedExistsSubquery)


Unrelated change?

Oh, nvm, I saw it.

viirya · 2020-01-05T18:09:54Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala

@@ -64,9 +64,10 @@ object SubqueryExpression {
  /**
   * Returns true when an expression contains an IN or EXISTS subquery and false otherwise.


We should update the doc too.

viirya · 2020-01-05T18:17:51Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala

@@ -52,6 +52,21 @@ object ReplaceExpressions extends Rule[LogicalPlan] {
  }
 }

+/**
+ * Rewrite non correlated exists subquery to use ScalarSubquery


non correlated -> uncorrelated

viirya · 2020-01-05T18:24:33Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala

+ * will be rewrite to
+ *   WHERE (SELECT 1 FROM (SELECT A FROM TABLE B WHERE COL1 > 10) LIMIT 1) IS NOT NULL
+ */
+object RewriteNonCorrelatedExists extends Rule[LogicalPlan] {


Shall we add a test for this rule?

Shall we add a test for this rule?

With test case

test("Rewritten uncorrelated exists subquery to use ScalarSubquery") { val relation = LocalRelation('a.int) val relExistSubquery = LocalRelation('x.int, 'y.int, 'z.int).where('x > 10) val query = relation.where(Exists(relExistSubquery)).select('a) val optimized = Optimize.execute(query.analyze) val correctAnswer = relation .where(IsNotNull(ScalarSubquery(Limit(Literal(1), Project(Seq(Alias(Literal(1), "col")()), relExistSubquery))))) .analyze comparePlans(optimized, correctAnswer) }

Get error

\[info] RewriteSubquerySuite: [info] - Rewritten uncorrelated exists subquery to use ScalarSubquery *** FAILED *** (852 milliseconds) [info] == FAIL: Plans do not match === [info] Filter isnotnull(scalar-subquery#0 []) Filter isnotnull(scalar-subquery#0 []) [info] : +- GlobalLimit 1 : +- GlobalLimit 1 [info] : +- LocalLimit 1 : +- LocalLimit 1 [info] !: +- Project [1 AS col#5] : +- Project [1 AS col#6] [info] : +- Filter (x#1 > 10) : +- Filter (x#1 > 10) [info] : +- LocalRelation <empty>, [x#1, y#2, z#3] : +- LocalRelation <empty>, [x#1, y#2, z#3] [info] +- LocalRelation <empty>, [a#0] +- LocalRelation <empty>, [a#0] (PlanTest.scala:147) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)

Because of Alias in RewriteNonCorrelatedExists .

Any good advise for test case, where I add test case can avoid this problem? @cloud-fan @viirya

viirya

I also noticed that in related docs (in Spark code or not), Exists expression seems to be correlated condition.

For example, the Exists doc said The [[Exists]] expression checks if a row exists in a subquery given some correlated condition..

Although uncorrelated exist query definitely work now and we also have such query in tests like exist-basic.sql.

Maybe we should also update the doc.

SparkQA · 2020-01-05T21:55:27Z

Test build #116119 has finished for PR 26437 at commit 9a9d9d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-01-06T03:23:49Z

I also noticed that in related docs (in Spark code or not), Exists expression seems to be correlated condition.

For example, the Exists doc said The [[Exists]] expression checks if a row exists in a subquery given some correlated condition..

Although uncorrelated exist query definitely work now and we also have such query in tests like exist-basic.sql.

Maybe we should also update the doc.

Updated what you have mentioned.

cloud-fan · 2020-01-06T03:54:36Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala

@@ -52,6 +52,21 @@ object ReplaceExpressions extends Rule[LogicalPlan] {
  }
 }

+/**
+ * Rewritten uncorrelated exists subquery to use ScalarSubquery


Rewrite non-correlated

Rewrite non-correlated

updated.

SparkQA · 2020-01-06T05:38:59Z

Test build #4986 has finished for PR 26437 at commit 26258b0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-01-06T06:55:42Z

retest this please

SparkQA · 2020-01-06T08:05:02Z

Test build #116134 has finished for PR 26437 at commit 26258b0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-01-06T08:07:29Z

retest this please

SparkQA · 2020-01-06T12:09:27Z

Test build #116144 has finished for PR 26437 at commit 26258b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-01-06T14:54:53Z

thanks, merging to master!

AngersZhuuuu added 3 commits November 8, 2019 18:26

plan exists subquery

2d762b4

Update subquery.scala

1c577bc

format import

5fa971b

cloud-fan reviewed Nov 8, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala Outdated Show resolved Hide resolved

AngersZhuuuu added 2 commits November 8, 2019 22:29

don;t collect executed rdd

1401349

format code

7b943aa

cloud-fan reviewed Nov 8, 2019

View reviewed changes

dongjoon-hyun added the SQL label Nov 9, 2019

AngersZhuuuu added 2 commits November 10, 2019 16:26

Update predicates.scala

95e446d

Update subquery.scala

20cda42

AngersZhuuuu commented Nov 10, 2019

View reviewed changes

remove ExistsSubquery

8e3ce4f

minimize cost

c290411

AngersZhuuuu changed the title ~~[SPARK-29800][SQL] Plan Exists 's subquery in PlanSubqueries~~ [SPARK-29800][SQL] Plan non-correlated Exists 's subquery in PlanSubqueries Nov 11, 2019

cloud-fan reviewed Nov 11, 2019

View reviewed changes

fix comment error

9a9d9d1

viirya reviewed Jan 5, 2020

View reviewed changes

follow comment

173942d

cloud-fan reviewed Jan 6, 2020

View reviewed changes

Update finishAnalysis.scala

26258b0

AngersZhuuuu changed the title ~~[SPARK-29800][SQL] Rewrite non-correlated subquery use ScalaSubquery to optimize perf~~ [SPARK-29800][SQL] Rewrite non-correlated EXISTS subquery use ScalaSubquery to optimize perf Jan 6, 2020

cloud-fan closed this in 3eade74 Jan 6, 2020

AngersZhuuuu mentioned this pull request Jan 7, 2020

[SPARK-29769][SQL] Support non-correlate "exists/not exists" condition as all type Join's on condition #26431

Closed



		def updateResult(): Unit = {
		val rows = plan.executeCollect()



		def updateResult(): Unit = {
		result = !plan.execute().isEmpty()

		splitConjunctivePredicates(condition)
		.partition(SubqueryExpression.hasInOrCorrelatedExistsSubquery)

		@@ -64,9 +64,10 @@ object SubqueryExpression {
		/**
		* Returns true when an expression contains an IN or EXISTS subquery and false otherwise.

[SPARK-29800][SQL] Rewrite non-correlated EXISTS subquery use ScalaSubquery to optimize perf #26437

[SPARK-29800][SQL] Rewrite non-correlated EXISTS subquery use ScalaSubquery to optimize perf #26437

Conversation

AngersZhuuuu commented Nov 8, 2019 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

AngersZhuuuu commented Nov 8, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dilipbiswal Nov 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AngersZhuuuu Nov 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dilipbiswal Nov 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AngersZhuuuu commented Nov 10, 2019 • edited Loading

cloud-fan commented Nov 11, 2019

AngersZhuuuu commented Nov 11, 2019

cloud-fan commented Nov 11, 2019

AngersZhuuuu commented Nov 11, 2019

AngersZhuuuu commented Nov 11, 2019

cloud-fan commented Nov 11, 2019

AngersZhuuuu commented Nov 11, 2019 • edited Loading

cloud-fan commented Nov 11, 2019

AngersZhuuuu commented Nov 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 5, 2020

viirya commented Jan 5, 2020

AngersZhuuuu commented Jan 5, 2020 • edited Loading

viirya commented Jan 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya left a comment

Choose a reason for hiding this comment

SparkQA commented Jan 5, 2020

AngersZhuuuu commented Jan 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 6, 2020

cloud-fan commented Jan 6, 2020

SparkQA commented Jan 6, 2020

cloud-fan commented Jan 6, 2020

SparkQA commented Jan 6, 2020

cloud-fan commented Jan 6, 2020

AngersZhuuuu commented Nov 8, 2019 •

edited

Loading

dilipbiswal Nov 8, 2019 •

edited

Loading

AngersZhuuuu Nov 9, 2019 •

edited

Loading

dilipbiswal Nov 11, 2019 •

edited

Loading

AngersZhuuuu commented Nov 10, 2019 •

edited

Loading

AngersZhuuuu commented Nov 11, 2019 •

edited

Loading

AngersZhuuuu commented Nov 11, 2019 •

edited

Loading

AngersZhuuuu commented Jan 5, 2020 •

edited

Loading