Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-34012][SQL][2.4] Keep behavior consistent when conf spark.sqllegacy.parser.havingWithoutGroupByAsWhere is true with migration guide #31050

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -467,7 +467,11 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
val withProject = if (aggregation == null && having != null) {
if (conf.getConf(SQLConf.LEGACY_HAVING_WITHOUT_GROUP_BY_AS_WHERE)) {
// If the legacy conf is set, treat HAVING without GROUP BY as WHERE.
withHaving(having, createProject())
val predicate = expression(having) match {
case p: Predicate => p
case e => Cast(e, BooleanType)
}
Filter(predicate, createProject())
} else {
// According to SQL standard, HAVING without GROUP BY means global aggregate.
withHaving(having, Aggregate(Nil, namedExpressions, withFilter))
Expand Down
10 changes: 10 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/group-by.sql
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,13 @@ SELECT 1 FROM range(10) HAVING true;
SELECT 1 FROM range(10) HAVING MAX(id) > 0;

SELECT id FROM range(10) HAVING id > 0;

SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=true;

SELECT 1 FROM range(10) HAVING true;

SELECT 1 FROM range(10) HAVING MAX(id) > 0;

SELECT id FROM range(10) HAVING id > 0;

SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=false;
60 changes: 59 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/group-by.sql.out
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 30
-- Number of queries: 35


-- !query 0
Expand Down Expand Up @@ -275,3 +275,61 @@ struct<>
-- !query 29 output
org.apache.spark.sql.AnalysisException
grouping expressions sequence is empty, and '`id`' is not an aggregate function. Wrap '()' in windowing function(s) or wrap '`id`' in first() (or first_value) if you don't care which value you get.;


-- !query 30
SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=true
-- !query 30 schema
struct<key:string,value:string>
-- !query 30 output
spark.sql.legacy.parser.havingWithoutGroupByAsWhere true


-- !query 31
SELECT 1 FROM range(10) HAVING true
-- !query 31 schema
struct<1:int>
-- !query 31 output
1
1
1
1
1
1
1
1
1
1


-- !query 32
SELECT 1 FROM range(10) HAVING MAX(id) > 0
-- !query 32 schema
struct<>
-- !query 32 output
java.lang.UnsupportedOperationException
Cannot evaluate expression: max(input[0, bigint, false])


-- !query 33
SELECT id FROM range(10) HAVING id > 0
-- !query 33 schema
struct<id:bigint>
-- !query 33 output
1
2
3
4
5
6
7
8
9


-- !query 34
SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=false
-- !query 34 schema
struct<key:string,value:string>
-- !query 34 output
spark.sql.legacy.parser.havingWithoutGroupByAsWhere false