[SPARK-21354] [SQL] INPUT FILE related functions do not support more than one sources #18580

gatorsmile · 2017-07-09T23:03:02Z

What changes were proposed in this pull request?

The build-in functions input_file_name, input_file_block_start, input_file_block_length do not support more than one sources, like what Hive does. Currently, Spark does not block it and the outputs are ambiguous/non-deterministic. It could be from any side.

hive> select *, INPUT__FILE__NAME FROM t1, t2;
FAILED: SemanticException Column INPUT__FILE__NAME Found in more than One Tables/Subqueries

This PR blocks it and issues an error.

How was this patch tested?

Added a test case

SparkQA · 2017-07-10T01:29:54Z

Test build #79426 has finished for PR 18580 at commit 596ea17.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-07-10T01:47:25Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala

@@ -100,6 +104,10 @@ trait CheckAnalysis extends PredicateHelper {
            failAnalysis(
              s"invalid cast from ${c.child.dataType.simpleString} to ${c.dataType.simpleString}")

+          case e @ (_: InputFileName | _: InputFileBlockLength | _: InputFileBlockStart)
+              if getNumLeafNodes(operator) > 1 =>
+            e.failAnalysis(s"'${e.prettyName}' does not support more than one sources")


nit: one sources -> one source.

dongjoon-hyun · 2017-07-10T01:47:34Z

sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala

+        val e = intercept[AnalysisException] {
+          sql(s"SELECT *, $func() FROM tab1 JOIN tab2 ON tab1.id = tab2.id")
+        }.getMessage
+        assert(e.contains(s"'$func' does not support more than one sources"))


dongjoon-hyun

.

dongjoon-hyun

Opps. Sorry. It seems we need to consider UNION ALL. UNION ALL works correctly before.

sql("select *, input_file_name() from ((select * from tab1) union all (select * from tab2))").show(false)

gatorsmile · 2017-07-10T07:28:41Z

@dongjoon-hyun What is the output of Hive for your case?

dongjoon-hyun · 2017-07-10T07:37:11Z

The following is the output of the current Spark.

scala> spark.range(10).write.saveAsTable("t1")

scala> spark.range(100,110).write.saveAsTable("t2")

scala> sql("select *, input_file_name() from t1").show(false)
+---+-------------------------------------------------------------------------------------------------------------------+
|id |input_file_name()                                                                                                  |
+---+-------------------------------------------------------------------------------------------------------------------+
|3  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|4  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|8  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|9  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|0  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00000-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|1  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00001-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|2  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00002-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|5  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00004-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|6  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00005-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|7  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00006-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
+---+-------------------------------------------------------------------------------------------------------------------+


scala> sql("select *, input_file_name() from t2").show(false)
+---+-------------------------------------------------------------------------------------------------------------------+
|id |input_file_name()                                                                                                  |
+---+-------------------------------------------------------------------------------------------------------------------+
|103|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00003-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|104|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00003-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|108|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00007-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|109|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00007-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|100|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00000-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|101|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00001-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|102|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00002-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|105|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00004-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|106|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00005-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|107|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00006-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
+---+-------------------------------------------------------------------------------------------------------------------+


scala> sql("select *, input_file_name() from ((select * from t1) union all (select * from t2))").show(false)
+---+-------------------------------------------------------------------------------------------------------------------+
|id |input_file_name()                                                                                                  |
+---+-------------------------------------------------------------------------------------------------------------------+
|3  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|4  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|8  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|9  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|0  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00000-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|1  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00001-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|2  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00002-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|5  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00004-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|6  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00005-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|7  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00006-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|103|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00003-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|104|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00003-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|108|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00007-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|109|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00007-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|100|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00000-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|101|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00001-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|102|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00002-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|105|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00004-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|106|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00005-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|107|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00006-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
+---+-------------------------------------------------------------------------------------------------------------------+

dongjoon-hyun · 2017-07-10T07:38:06Z

In case of UNION ALL, everything looks normal. So, it would be a case of regression of Spark.

gatorsmile · 2017-07-10T17:58:58Z

Does Hive report an error?

dongjoon-hyun · 2017-07-10T18:12:22Z

Let me check that. BTW, I think Spark is better than Hive. :)

dongjoon-hyun · 2017-07-10T18:24:00Z

This is the error and difference.

HIVE

hive> select *, INPUT__FILE__NAME from (select * from t1) T;
FAILED: SemanticException [Error 10004]: Line 1:10 Invalid table alias or column reference 'INPUT__FILE__NAME': (possible column names are: _c0)

SPARK

scala> sql("select *, input_file_name() from (select * from t1)").show(false)
+---+-------------------------------------------------------------------------------------------------------------------+
|id |input_file_name()                                                                                                  |
+---+-------------------------------------------------------------------------------------------------------------------+
|3  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|4  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|8  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|9  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|0  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00000-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|1  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00001-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|2  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00002-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|5  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00004-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|6  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00005-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|7  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00006-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
+---+-------------------------------------------------------------------------------------------------------------------+

gatorsmile · 2017-07-10T18:55:08Z

For union, does Hive output an error?

dongjoon-hyun · 2017-07-10T19:07:47Z

It was the same error, Invalid table alias or column reference 'INPUT__FILE__NAME': (possible column names are: _c0). So, I reduce that into a simple above example.

hive> select *, INPUT__FILE__NAME from (select * from t1 union all select * from t1) T;
FAILED: SemanticException [Error 10004]: Line 1:10 Invalid table alias or column reference 'INPUT__FILE__NAME': (possible column names are: _c0)

gatorsmile · 2017-07-11T04:17:42Z

cc @cloud-fan

cloud-fan · 2017-07-11T05:33:15Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala

@@ -74,6 +74,15 @@ trait CheckAnalysis extends PredicateHelper {
    }
  }

+  private def getNumInputFileBlockSources(operator: LogicalPlan): Int = {
+    operator match {
+      case _: LeafNode => 1


shall we only consider file data source leaf node?

Unable to check it in CheckAnalysis. Both HadoopRDD and FileScanRDD have the same issues. To block both, we need to add the check as another rule.

SparkQA · 2017-07-11T05:36:40Z

Test build #79496 has finished for PR 18580 at commit 6b48a9e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-07-13T03:50:14Z

Retest this please.

SparkQA · 2017-07-13T06:19:12Z

Test build #79576 has finished for PR 18580 at commit 6b48a9e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-07-17T02:32:45Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala

+      case o =>
+        val numInputFileBlockSources = o.children.map(checkNumInputFileBlockSources(e, _)).sum
+        if (numInputFileBlockSources > 1) {
+          e.failAnalysis(s"'${e.prettyName}' does not support more than one sources")


Need to check it as early as possible; otherwise, Union might eat it.

cloud-fan · 2017-07-17T04:56:52Z

LGTM, pending jenkins

SparkQA · 2017-07-17T05:07:25Z

Test build #79651 has finished for PR 18580 at commit c4de2b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-07-17T06:58:28Z

thanks, merging to master!

gatorsmile added 2 commits July 9, 2017 15:52

fix.

04ff406

fix.

596ea17

dongjoon-hyun reviewed Jul 10, 2017

View reviewed changes

dongjoon-hyun approved these changes Jul 10, 2017

View reviewed changes

dongjoon-hyun reviewed Jul 10, 2017

View reviewed changes

fix.

6b48a9e

cloud-fan reviewed Jul 11, 2017

View reviewed changes

fix.

c4de2b8

gatorsmile commented Jul 17, 2017

View reviewed changes

asfgit closed this in e398c28 Jul 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-21354] [SQL] INPUT FILE related functions do not support more than one sources #18580

[SPARK-21354] [SQL] INPUT FILE related functions do not support more than one sources #18580

gatorsmile commented Jul 9, 2017

SparkQA commented Jul 10, 2017

dongjoon-hyun Jul 10, 2017

dongjoon-hyun Jul 10, 2017

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

gatorsmile commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017 •

edited

Loading

gatorsmile commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017 •

edited

Loading

gatorsmile commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017 •

edited

Loading

gatorsmile commented Jul 11, 2017

cloud-fan Jul 11, 2017

gatorsmile Jul 17, 2017

SparkQA commented Jul 11, 2017

dongjoon-hyun commented Jul 13, 2017

SparkQA commented Jul 13, 2017

gatorsmile Jul 17, 2017

cloud-fan commented Jul 17, 2017

SparkQA commented Jul 17, 2017

cloud-fan commented Jul 17, 2017

[SPARK-21354] [SQL] INPUT FILE related functions do not support more than one sources #18580

[SPARK-21354] [SQL] INPUT FILE related functions do not support more than one sources #18580

Conversation

gatorsmile commented Jul 9, 2017

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Jul 10, 2017

dongjoon-hyun Jul 10, 2017

Choose a reason for hiding this comment

dongjoon-hyun Jul 10, 2017

Choose a reason for hiding this comment

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

gatorsmile commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017 • edited Loading

gatorsmile commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017 • edited Loading

gatorsmile commented Jul 10, 2017

dongjoon-hyun commented Jul 10, 2017 • edited Loading

gatorsmile commented Jul 11, 2017

cloud-fan Jul 11, 2017

Choose a reason for hiding this comment

gatorsmile Jul 17, 2017

Choose a reason for hiding this comment

SparkQA commented Jul 11, 2017

dongjoon-hyun commented Jul 13, 2017

SparkQA commented Jul 13, 2017

gatorsmile Jul 17, 2017

Choose a reason for hiding this comment

cloud-fan commented Jul 17, 2017

SparkQA commented Jul 17, 2017

cloud-fan commented Jul 17, 2017

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Jul 10, 2017 •

edited

Loading

dongjoon-hyun commented Jul 10, 2017 •

edited

Loading

dongjoon-hyun commented Jul 10, 2017 •

edited

Loading