Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix tests failures in window_function_test.py #11019

Closed
Tracked by #11004
razajafri opened this issue Jun 8, 2024 · 1 comment · Fixed by #11073
Closed
Tracked by #11004

Fix tests failures in window_function_test.py #11019

razajafri opened this issue Jun 8, 2024 · 1 comment · Fixed by #11073
Assignees
@mythrocks
Labels
bug Something isn't working Spark 4.0+ Spark 4.0+ issues

Comments

@razajafri
Copy link
Collaborator 

FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_decimal128_count_window
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_decimal128_count_window_no_part
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_decimal_running_sum_window
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_decimal_running_sum_window_no_part
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_decimal_sum_window
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_decimal_sum_window_no_part
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_join_sum_window_of_window
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_multi_types_window_aggs_for_rows
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_multi_types_window_aggs_for_rows_lead_lag
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_nested_part_struct
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_range_running_window_float_decimal_sum_runs_batched
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_range_running_window_runs_batched
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_range_windows_with_string_order_by_column
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_rows_based_running_window_partitioned
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_rows_based_running_window_unpartitioned
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_running_window_function_exec_for_all_aggs
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_running_window_without_partitions_runs_batched
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_to_date_with_window_functions
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_unbounded_to_unbounded_window
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggregations_for_big_decimal_ranges
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggregations_for_decimal_and_float_ranges
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggs_for_batched_finite_row_windows_fallback
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggs_for_batched_finite_row_windows_partitioned
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggs_for_batched_finite_row_windows_unpartitioned
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggs_for_negative_rows_partitioned
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggs_for_negative_rows_unpartitioned
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggs_for_range_numeric_date
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggs_for_ranges_timestamps
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_aggs_for_rows
FAILED ../../../../integration_tests/src/main/python/window_function_test.py::test_window_running_float_decimal_sum
@razajafri razajafri added bug Something isn't working ? - Needs Triage Need team to review and classify labels Jun 8, 2024
@razajafri razajafri mentioned this issue Jun 8, 2024
Open
@razajafri razajafri added the Spark 4.0+ Spark 4.0+ issues label Jun 8, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Jun 11, 2024
@mythrocks mythrocks self-assigned this Jun 11, 2024
@mythrocks
Copy link
Collaborator

mythrocks commented Jun 11, 2024 •
edited
Loading

This was the one I was most worried about. It looks like the tests pass correctly when spark.sql.ansi.enabled=false.

I'm inclined to have spark.sql.ansi.enabled=false for these tests, and have explicit tests for ANSI mode enabled, but I haven't completely thought it through.

mythrocks added a commit to mythrocks/spark-rapids that referenced this issue Jun 17, 2024
@mythrocks
Disable ANSI mode for window function tests.
3db464b 
Fixes NVIDIA#11019.

Window function tests fail on Spark 4.0 because of NVIDIA#5114 (and NVIDIA#5120 broadly),
because spark-rapids does not support SUM, COUNT, and certain other aggregations
in ANSI mode.

This commit disables ANSI mode tests for the failing window function tests. These may be
revisited, once error/overflow checking is available for ANSI mode in spark-rapids.

Signed-off-by: MithunR <mithunr@nvidia.com>
@mythrocks mythrocks mentioned this issue Jun 17, 2024
Merged
wjxiz1992 added a commit to nvliyuan/yuali-spark-rapids that referenced this issue Jun 26, 2024
@wjxiz1992 @binmahone
@razajafri @jihoonson @gerashegalov @mythrocks
optimzing Expand+Aggregate in sqlw with many count distinct (#22)
4415f61 
* optimzing Expand+Aggregate in sqlw with many count distinct

Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>

* Add GpuBucketingUtils shim to Spark 4.0.0 (NVIDIA#11092)

* Add GpuBucketingUtils shim to Spark 4.0.0

* Signing off

Signed-off-by: Raza Jafri <rjafri@nvidia.com>

---------

Signed-off-by: Raza Jafri <rjafri@nvidia.com>

* Improve the diagnostics for 'conv' fallback explain (NVIDIA#11076)

* Improve the diagnostics for 'conv' fallback explain

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

* don't use nil

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

* the bases should not be an empty string in the error message when the user input is not

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

* more user-friendly message

* Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala

Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>

---------

Signed-off-by: Jihoon Son <ghoonson@gmail.com>
Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>

* Disable ANSI mode for window function tests [databricks] (NVIDIA#11073)

* Disable ANSI mode for window function tests.

Fixes NVIDIA#11019.

Window function tests fail on Spark 4.0 because of NVIDIA#5114 (and NVIDIA#5120 broadly),
because spark-rapids does not support SUM, COUNT, and certain other aggregations
in ANSI mode.

This commit disables ANSI mode tests for the failing window function tests. These may be
revisited, once error/overflow checking is available for ANSI mode in spark-rapids.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Switch from @ansi_mode_disabled to @disable_ansi_mode.

---------

Signed-off-by: MithunR <mithunr@nvidia.com>

---------

Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
Signed-off-by: Jihoon Son <ghoonson@gmail.com>
Signed-off-by: MithunR <mithunr@nvidia.com>
Co-authored-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
Co-authored-by: Raza Jafri <razajafri@users.noreply.github.com>
Co-authored-by: Jihoon Son <jihoonson@apache.org>
Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>
Co-authored-by: MithunR <mithunr@nvidia.com>
SurajAralihalli pushed a commit to SurajAralihalli/spark-rapids that referenced this issue Jul 12, 2024
@mythrocks @SurajAralihalli
Disable ANSI mode for window function tests [databricks] (NVIDIA#11073)
11348ef 
* Disable ANSI mode for window function tests.

Fixes NVIDIA#11019.

Window function tests fail on Spark 4.0 because of NVIDIA#5114 (and NVIDIA#5120 broadly),
because spark-rapids does not support SUM, COUNT, and certain other aggregations
in ANSI mode.

This commit disables ANSI mode tests for the failing window function tests. These may be
revisited, once error/overflow checking is available for ANSI mode in spark-rapids.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Switch from @ansi_mode_disabled to @disable_ansi_mode.

---------

Signed-off-by: MithunR <mithunr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mythrocks mythrocks

Labels
bug Something isn't working Spark 4.0+ Spark 4.0+ issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Disable ANSI mode for window function tests [databricks] mythrocks/spark-rapids
3 participants
@mattahrens @mythrocks @razajafri