-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-21787][SQL] Support for pushing down filters for DateType in native OrcFileFormat #18995
Conversation
Hi, @cloud-fan, @gatorsmile, @rxin , @sameeragarwal . |
Test build #80854 has finished for PR 18995 at commit
|
Test build #80857 has finished for PR 18995 at commit
|
This will be supported in a new ORCFileFormat after merging #18953. |
Test build #83700 has finished for PR 18995 at commit
|
|
||
import scala.collection.JavaConverters._ | ||
|
||
import org.apache.orc.storage.ql.io.sarg.{PredicateLeaf, SearchArgument} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please note that this suite use Apache ORC SearchArgument
and PredicateLeaf
classes.
We cannot share the test code for this suite.
import testImplicits._ | ||
|
||
protected def format: String |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By using full canonical class names, we will explicitly test new OrcFileFormat in sql/core
and old OrcFileFormat in sql/hive
without considering SQL ORC switching conf.
Hi, @cloud-fan . |
Test build #84405 has finished for PR 18995 at commit
|
maybe have a PR to move the tests first? |
For the other part, I'm restructuring to remove redundancy in the same manner of This one cannot be the part of that restructuring, because the test function signatures are different due to Could you review this first? |
BTW, @cloud-fan . Do you mean literally Otherwise, I had a set of ported duplicated test suites in the previous PRs. Is it okay? |
Test build #84609 has finished for PR 18995 at commit
|
Hi, @cloud-fan . |
thanks, merging to master! |
Thank you so much, @cloud-fan ! |
What changes were proposed in this pull request?
This PR support for pushing down filters for DateType in ORC
How was this patch tested?
Pass the Jenkins with newly add and updated test cases.