-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Spark raise_error function #10110
Add Spark raise_error function #10110
Conversation
✅ Deploy Preview for meta-velox canceled.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
ping @rui-mo |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you run the fuzzer test for this function to make sure it works? Thanks.
I have verified locally end to end with gluten, @rui-mo what do you means for |
@gaoyangxiaozhu The command is like below, and you could use |
@rui-mo any new comments ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks you.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Looks good overall.
@gaoyangxiaozhu Could you try the fuzzer test again to make sure it works? Thanks. |
|
@gaoyangxiaozhu I assume we need to rebase to #10274 and returns status in raise_error function after it gets merged. Thanks. |
sure, so could you help sign the PR or we need people from meta to review and help sign ? |
@gaoyangxiaozhu Will continue to review this PR after the code is rebased to #10274. Thanks. |
@gaoyangxiaozhu #10274 has been merged. Could you rebase to it and update the code? Thanks. |
Done, and also local test with fuzze test. should be fine. @rui-mo |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have no other comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
@bikramSingh91 merged this pull request in 4e39b06. |
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
This PR adds Spark
raise_error
function. It is re-used in Sparkassert_true
function, so after this PR
assert_true
can also be supported.Spark document: https://spark.apache.org/docs/latest/api/sql/#raise_error
Spark code: https://github.com/apache/spark/blob/branch-3.5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala#L85C1-L92C1
Issue: apache/incubator-gluten#5991