[SPARK-18016][SQL][FOLLOW-UP] Code Generation: Constant Pool Limit - reduce entries for mutable state #20036

kiszk · 2017-12-20T16:26:28Z

What changes were proposed in this pull request?

This PR addresses additional review comments in #19811

How was this patch tested?

Existing test suites

SparkQA · 2017-12-20T18:54:56Z

Test build #85196 has finished for PR 20036 at commit 53661eb.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2017-12-21T01:23:04Z

Jenkins, retest this please

SparkQA · 2017-12-21T03:57:17Z

Test build #85224 has finished for PR 20036 at commit 53661eb.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2017-12-21T05:46:41Z

Jenkins, retest this please

SparkQA · 2017-12-21T08:05:01Z

Test build #85236 has finished for PR 20036 at commit 53661eb.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-12-21T11:54:34Z

retest this please

SparkQA · 2017-12-21T14:50:05Z

Test build #85260 has finished for PR 20036 at commit 53661eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2017-12-22T05:13:22Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala

        val pattern = ctx.addMutableState(patternClass, "patternLike",
-          v => s"""$v = ${patternClass}.compile("$regexStr");""", forceInline = true)
+          v => s"""$v = ${patternClass}.compile("$regexStr");""")


nit: we can remove { and } around patternClass.

Sure, done for other places, too.

ueshin · 2017-12-22T05:13:30Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala

        val pattern = ctx.addMutableState(patternClass, "patternRLike",
-          v => s"""$v = ${patternClass}.compile("$regexStr");""", forceInline = true)
+          v => s"""$v = ${patternClass}.compile("$regexStr");""")


SparkQA · 2017-12-22T08:05:01Z

Test build #85300 has finished for PR 20036 at commit 8d8b669.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2017-12-22T08:09:57Z

retest this please

SparkQA · 2017-12-22T09:54:22Z

Test build #85301 has finished for PR 20036 at commit 8d8b669.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2017-12-22T10:12:53Z

retest this please

SparkQA · 2017-12-22T13:18:02Z

Test build #85307 has finished for PR 20036 at commit 8d8b669.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2017-12-26T08:33:21Z

ping @cloud-fan

cloud-fan · 2017-12-27T01:48:48Z

sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala

@@ -283,7 +283,7 @@ case class InputAdapter(child: SparkPlan) extends UnaryExecNode with CodegenSupp

  override def doProduce(ctx: CodegenContext): String = {
    // Right now, InputAdapter is only used when there is one input RDD.
-    // inline mutable state since an inputAdaptor in a task
+    // inline mutable state since an InputAdapter in a task


you miss some words...

cloud-fan · 2017-12-27T02:04:50Z

...atalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala

@@ -299,7 +299,7 @@ class CodegenContext {
  def initMutableStates(): String = {
    // It's possible that we add same mutable state twice, e.g. the `mergeExpressions` in
    // `TypedAggregateExpression`, we should call `distinct` here to remove the duplicated ones.
-    val initCodes = mutableStateInitCode.distinct
+    val initCodes = mutableStateInitCode.distinct.map(_ + "\n")


ah, good catch!

cloud-fan · 2017-12-27T02:05:30Z

LGTM

SparkQA · 2017-12-27T07:50:21Z

Test build #85423 has finished for PR 20036 at commit 05da9d7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2017-12-27T08:06:54Z

sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala

@@ -283,7 +283,7 @@ case class InputAdapter(child: SparkPlan) extends UnaryExecNode with CodegenSupp

  override def doProduce(ctx: CodegenContext): String = {
    // Right now, InputAdapter is only used when there is one input RDD.
-    // inline mutable state since an inputAdaptor in a task
+    // inline mutable state since an InputAdapter is used once in a task for WholeStageCodegen


nit: inline -> Inline

viirya · 2017-12-27T08:08:47Z

sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala

@@ -440,8 +440,9 @@ case class SortMergeJoinExec(
    val spillThreshold = getSpillThreshold
    val inMemoryThreshold = getInMemoryThreshold

+    // inline mutable state since not many join operations in a task


nit: Inline

viirya · 2017-12-27T08:08:52Z

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala

      } else {
        val generatedMap = new RowBasedHashMapGenerator(ctx, aggregateExpressions,
          fastHashMapClassName, groupingKeySchema, bufferSchema).generate()
        ctx.addInnerClass(generatedMap)

+        // inline mutable state since not many aggregation operations in a task


nit: Inline

viirya · 2017-12-27T08:08:56Z

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala

@@ -587,20 +587,24 @@ case class HashAggregateExec(
          fastHashMapClassName, groupingKeySchema, bufferSchema).generate()
        ctx.addInnerClass(generatedMap)

+        // inline mutable state since not many aggregation operations in a task


nit: Inline

viirya · 2017-12-27T08:09:39Z

LGTM

SparkQA · 2017-12-27T11:40:16Z

Test build #85430 has finished for PR 20036 at commit 4ef81c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-12-28T04:28:32Z

thanks, merging to master!

initial commit

53661eb

ueshin reviewed Dec 22, 2017

View reviewed changes

address review comments

8d8b669

cloud-fan reviewed Dec 27, 2017

View reviewed changes

address review comment

05da9d7

viirya reviewed Dec 27, 2017

View reviewed changes

address review comment

4ef81c8

asfgit closed this in 5683984 Dec 28, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-18016][SQL][FOLLOW-UP] Code Generation: Constant Pool Limit - reduce entries for mutable state #20036

[SPARK-18016][SQL][FOLLOW-UP] Code Generation: Constant Pool Limit - reduce entries for mutable state #20036

kiszk commented Dec 20, 2017

SparkQA commented Dec 20, 2017

kiszk commented Dec 21, 2017

SparkQA commented Dec 21, 2017

kiszk commented Dec 21, 2017

SparkQA commented Dec 21, 2017

HyukjinKwon commented Dec 21, 2017

SparkQA commented Dec 21, 2017

ueshin Dec 22, 2017

kiszk Dec 22, 2017

ueshin Dec 22, 2017

SparkQA commented Dec 22, 2017

kiszk commented Dec 22, 2017

SparkQA commented Dec 22, 2017

kiszk commented Dec 22, 2017

SparkQA commented Dec 22, 2017

kiszk commented Dec 26, 2017

cloud-fan Dec 27, 2017

kiszk Dec 27, 2017

cloud-fan Dec 27, 2017

cloud-fan commented Dec 27, 2017

SparkQA commented Dec 27, 2017

viirya Dec 27, 2017

viirya Dec 27, 2017

viirya Dec 27, 2017

viirya Dec 27, 2017

viirya commented Dec 27, 2017

SparkQA commented Dec 27, 2017

cloud-fan commented Dec 28, 2017

[SPARK-18016][SQL][FOLLOW-UP] Code Generation: Constant Pool Limit - reduce entries for mutable state #20036

[SPARK-18016][SQL][FOLLOW-UP] Code Generation: Constant Pool Limit - reduce entries for mutable state #20036

Conversation

kiszk commented Dec 20, 2017

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Dec 20, 2017

kiszk commented Dec 21, 2017

SparkQA commented Dec 21, 2017

kiszk commented Dec 21, 2017

SparkQA commented Dec 21, 2017

HyukjinKwon commented Dec 21, 2017

SparkQA commented Dec 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Dec 22, 2017

kiszk commented Dec 22, 2017

SparkQA commented Dec 22, 2017

kiszk commented Dec 22, 2017

SparkQA commented Dec 22, 2017

kiszk commented Dec 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Dec 27, 2017

SparkQA commented Dec 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya commented Dec 27, 2017

SparkQA commented Dec 27, 2017

cloud-fan commented Dec 28, 2017