Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-7294][SQL] ADD BETWEEN #5839

Closed
wants to merge 11 commits into from
Closed

[SPARK-7294][SQL] ADD BETWEEN #5839

wants to merge 11 commits into from

Conversation

kaka1992
Copy link
Contributor

@kaka1992 kaka1992 commented May 1, 2015

No description provided.

@kaka1992 kaka1992 changed the title [SPARK-7294] ADD BETWEEN [SPARK-7294][SQL] ADD BETWEEN May 1, 2015
rxin
rxin reviewed May 1, 2015
View reviewed changes
python/pyspark/sql/dataframe.py
[Row(col1=5, col2=6, col3=8)]
"""
#sc = SparkContext._active_spark_context
jc = self > col1 & self < col2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think u need to add parenthesis, i.e. (self > col1) & (self < col2)

rxin
rxin reviewed May 2, 2015
View reviewed changes
python/pyspark/sql/dataframe.py
[Row(col1=5, col2=6, col3=8)]
"""
jc = (self >= lowerBound) & (self <= upperBound)
return Column(jc)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually I think you no longer need to wrap it in Column, since it is already a Python column.

@rxin
Copy link
Contributor

rxin commented May 2, 2015

Jenkins, ok to test.

rxin
rxin reviewed May 2, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/Column.scala
* @group java_expr_ops
*/
def between(lowerBound: Column, upperBound: Column): Column = {
And(GreaterThanOrEqual(this.expr, lowerBound.expr),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually - how about we change the implementation here to use Column directly, i.e.

(this >= lowerBound) && (this <= upperBound)

and the Python one simply calls the JVM function?

@kaka1992
Copy link
Contributor Author

kaka1992 commented May 2, 2015

please test this pr.

@rxin
Copy link
Contributor

rxin commented May 3, 2015

Did you address my comments?

@kaka1992
Copy link
Contributor Author

kaka1992 commented May 4, 2015

@rxin Please test this RR.

rxin
rxin reviewed May 4, 2015
View reviewed changes
python/pyspark/sql/tests.py
Row(a=4, b=1, c=4)]).toDF()
self.assertEqual([False, True, True],
df.select(df.a.between(df.b, df.c)).collect())

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove one blank line.

@kaka1992
Copy link
Contributor Author

kaka1992 commented May 4, 2015

@rxin Please test this PR.

@rxin
Copy link
Contributor

rxin commented May 4, 2015

Jenkins, ok to test.

@SparkQA
Copy link

SparkQA commented May 4, 2015

Test build #31765 has finished for PR 5839 at commit 7e64d1e.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31819 has finished for PR 5839 at commit c54d904.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31822 has finished for PR 5839 at commit d2e7f72.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@kaka1992
Copy link
Contributor Author

kaka1992 commented May 5, 2015

Oro failure. Please retest it.

@rxin
Copy link
Contributor

rxin commented May 5, 2015

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31834 has finished for PR 5839 at commit d2e7f72.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31846 has finished for PR 5839 at commit f928816.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@kaka1992
Copy link
Contributor Author

kaka1992 commented May 5, 2015

Please retest this =_=. @rxin

@rxin
Copy link
Contributor

rxin commented May 5, 2015

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31891 has finished for PR 5839 at commit b15360d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented May 5, 2015

Thanks. I've merged this.

asfgit pushed a commit that referenced this pull request May 5, 2015
@rxin
[SPARK-7294][SQL] ADD BETWEEN
c68d0e2 
Author: 云峤 <chensong.cs@alibaba-inc.com>
Author: kaka1992 <kaka_1992@163.com>

Closes #5839 from kaka1992/master and squashes the following commits:

b15360d [kaka1992] Fix python unit test in sql/test. =_= I forget to commit this file last time.
f928816 [kaka1992] Fix python style in sql/test.
d2e7f72 [kaka1992] Fix python style in sql/test.
c54d904 [kaka1992] Fix empty map bug.
7e64d1e [云峤] Update
7b9b858 [云峤] undo
f080f8d [云峤] update pep8
76f0c51 [云峤] Merge remote-tracking branch 'remotes/upstream/master'
7d62368 [云峤] [SPARK-7294] ADD BETWEEN
baf839b [云峤] [SPARK-7294] ADD BETWEEN
d11d5b9 [云峤] [SPARK-7294] ADD BETWEEN

(cherry picked from commit 735bc3d)
Signed-off-by: Reynold Xin <rxin@databricks.com>
@asfgit asfgit closed this in 735bc3d May 5, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@jeanlyn
[SPARK-7294][SQL] ADD BETWEEN
8790ea6 
Author: 云峤 <chensong.cs@alibaba-inc.com>
Author: kaka1992 <kaka_1992@163.com>

Closes apache#5839 from kaka1992/master and squashes the following commits:

b15360d [kaka1992] Fix python unit test in sql/test. =_= I forget to commit this file last time.
f928816 [kaka1992] Fix python style in sql/test.
d2e7f72 [kaka1992] Fix python style in sql/test.
c54d904 [kaka1992] Fix empty map bug.
7e64d1e [云峤] Update
7b9b858 [云峤] undo
f080f8d [云峤] update pep8
76f0c51 [云峤] Merge remote-tracking branch 'remotes/upstream/master'
7d62368 [云峤] [SPARK-7294] ADD BETWEEN
baf839b [云峤] [SPARK-7294] ADD BETWEEN
d11d5b9 [云峤] [SPARK-7294] ADD BETWEEN
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@jeanlyn
[SPARK-7294][SQL] ADD BETWEEN
e99bc26 
Author: 云峤 <chensong.cs@alibaba-inc.com>
Author: kaka1992 <kaka_1992@163.com>

Closes apache#5839 from kaka1992/master and squashes the following commits:

b15360d [kaka1992] Fix python unit test in sql/test. =_= I forget to commit this file last time.
f928816 [kaka1992] Fix python style in sql/test.
d2e7f72 [kaka1992] Fix python style in sql/test.
c54d904 [kaka1992] Fix empty map bug.
7e64d1e [云峤] Update
7b9b858 [云峤] undo
f080f8d [云峤] update pep8
76f0c51 [云峤] Merge remote-tracking branch 'remotes/upstream/master'
7d62368 [云峤] [SPARK-7294] ADD BETWEEN
baf839b [云峤] [SPARK-7294] ADD BETWEEN
d11d5b9 [云峤] [SPARK-7294] ADD BETWEEN
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nemccarthy
[SPARK-7294][SQL] ADD BETWEEN
3b95ff6 
Author: 云峤 <chensong.cs@alibaba-inc.com>
Author: kaka1992 <kaka_1992@163.com>

Closes apache#5839 from kaka1992/master and squashes the following commits:

b15360d [kaka1992] Fix python unit test in sql/test. =_= I forget to commit this file last time.
f928816 [kaka1992] Fix python style in sql/test.
d2e7f72 [kaka1992] Fix python style in sql/test.
c54d904 [kaka1992] Fix empty map bug.
7e64d1e [云峤] Update
7b9b858 [云峤] undo
f080f8d [云峤] update pep8
76f0c51 [云峤] Merge remote-tracking branch 'remotes/upstream/master'
7d62368 [云峤] [SPARK-7294] ADD BETWEEN
baf839b [云峤] [SPARK-7294] ADD BETWEEN
d11d5b9 [云峤] [SPARK-7294] ADD BETWEEN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kaka1992 @rxin @SparkQA