Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-21787][SQL] Support for pushing down filters for DateType in native OrcFileFormat #18995

Closed
wants to merge 1 commit into from

Conversation

dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Aug 18, 2017

What changes were proposed in this pull request?

This PR support for pushing down filters for DateType in ORC

How was this patch tested?

Pass the Jenkins with newly add and updated test cases.

@dongjoon-hyun
Copy link
Member Author

Hi, @cloud-fan, @gatorsmile, @rxin , @sameeragarwal .
I tested this on #18953, this only works with Apache ORC 1.4.0.
This issue is reported saparately. And, to keep #18953 minimally, I didn't include this into that.

@SparkQA
Copy link

SparkQA commented Aug 18, 2017

Test build #80854 has finished for PR 18995 at commit 7f63202.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 18, 2017

Test build #80857 has finished for PR 18995 at commit b31acc5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-21787 branch November 2, 2017 18:11
@dongjoon-hyun dongjoon-hyun restored the SPARK-21787 branch November 10, 2017 19:28
@dongjoon-hyun dongjoon-hyun reopened this Nov 10, 2017
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC [WIP][SPARK-21787][SQL] Support for pushing down filters for DateType in ORC Nov 10, 2017
@dongjoon-hyun
Copy link
Member Author

This will be supported in a new ORCFileFormat after merging #18953.

@SparkQA
Copy link

SparkQA commented Nov 10, 2017

Test build #83700 has finished for PR 18995 at commit b31acc5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun changed the title [WIP][SPARK-21787][SQL] Support for pushing down filters for DateType in ORC [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC Dec 3, 2017

import scala.collection.JavaConverters._

import org.apache.orc.storage.ql.io.sarg.{PredicateLeaf, SearchArgument}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please note that this suite use Apache ORC SearchArgument and PredicateLeaf classes.
We cannot share the test code for this suite.

import testImplicits._

protected def format: String
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By using full canonical class names, we will explicitly test new OrcFileFormat in sql/core and old OrcFileFormat in sql/hive without considering SQL ORC switching conf.

@dongjoon-hyun
Copy link
Member Author

Hi, @cloud-fan .
This is the first followup after #19651 .
Could you review this PR, too?

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC [SPARK-21787][SPARK-22672][SQL] Support for pushing down filters for DateType in ORC Dec 3, 2017
@SparkQA
Copy link

SparkQA commented Dec 4, 2017

Test build #84405 has finished for PR 18995 at commit 9f6d75b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

maybe have a PR to move the tests first?

@dongjoon-hyun
Copy link
Member Author

@cloud-fan .

For the other part, I'm restructuring to remove redundancy in the same manner of InMemoryCatalogedDDLSuite, HiveCatalogedDDLSuite, and DDLSuite.

This one cannot be the part of that restructuring, because the test function signatures are different due to PredicateLeaf and SearchArgument.

Could you review this first?

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 4, 2017

BTW, @cloud-fan . Do you mean literally move? I'm wondering if I'm thinking in a different way.

Otherwise, I had a set of ported duplicated test suites in the previous PRs. Is it okay?

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-21787][SPARK-22672][SQL] Support for pushing down filters for DateType in ORC [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC Dec 7, 2017
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-21787][SQL] Support for pushing down filters for DateType in ORC [SPARK-21787][SQL] Support for pushing down filters for DateType in native OrcFileFormat Dec 7, 2017
@SparkQA
Copy link

SparkQA commented Dec 7, 2017

Test build #84609 has finished for PR 18995 at commit 18accdd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

Hi, @cloud-fan .
Could you review this PR again?

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@asfgit asfgit closed this in 0ba8f4b Dec 8, 2017
@dongjoon-hyun
Copy link
Member Author

Thank you so much, @cloud-fan !

@dongjoon-hyun dongjoon-hyun deleted the SPARK-21787 branch December 8, 2017 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants