Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: bump DataFusion to rev c6f0d3c #782

Closed
wants to merge 1 commit into from

Conversation

andygrove
Copy link
Member

Which issue does this PR close?

N/A

Rationale for this change

Weekly DataFusion revision bump.

What changes are included in this PR?

This bump includes the following DataFusion changes (generated with git log 35c2e7e..c6f0d3c --oneline):

c6f0d3cac (HEAD -> main, apache/main) Pass scalar to `eq` inside `nullif`  (#11697)
011a3f3e3 Improve readme page in crates.io (#11809)
f56a2ef28 doc: Add support for `map` and `make_map` functions (#11799)
0417e543e Minor: Update exected output due to logical conflict (#11824)
f19d30d52 chore(deps): update rstest requirement from 0.21.0 to 0.22.0 (#11811)
5c4254aeb Don't implement create_sliding_accumulator repeatedly (#11813)
45d85b1d1 Change name of MAX/MIN udaf to lowercase max/min  (#11795)
6aad19fad add valid distinct case for aggregate.slt (#11814)
682bc2eff Improve log func stability (#11808)
c340b6ab7 Skipping partial aggregation when it is not helping for high cardinality aggregates (#11627)
336c15e75 Improve MSRV CI check to print out problems to log (#11789)
1d3bdbe14 Minor: add ticket reference and fmt (#11805)
b4069a65a Remove `AggregateFunctionDefinition` (#11803)
c8e5996c4 Remove redundant Aggregate when `DISTINCT` & `GROUP BY` are in the same query (#11781)
a4d41d6a6 Support `LogicalPlan` `Debug` differently than `Display` (#11774)
f4e519f9d Move min and max to user defined aggregate function, remove `AggregateFunction` / `AggregateFunctionDefinition::BuiltIn` (#11013)
9e90e17a6 fix: Add additional required expression for natural join (#11713)
81668f3b2 Support planning `Map` literal (#11780)
6c4c24612 Doc: Add Sail to known users list (#11791)
de2da34cc Add docs and rename param (#11778)
b89037f0d Add references to github issue (#11784)
0332eb569 refactor: move ExecutionPlan and related structs into dedicated mod (#11759)
5ca4ec3b5 Extract CoalesceBatchesStream to a struct (#11610)
80848f2a0 Fix #11692: Improve doc comments within macros (#11694)
df4e6cc4e [Minor] Short circuit `ApplyFunctionRewrites` if there are no function rewrites (#11765)
d010ce90f refactor(11523): update OOM message provided for a single failed reservation (#11771)
70aba2bd6 minor: always time batch_filter even when the result is an empty batch (#11775)
a0ad37684 [Minor] Refactor approx_percentile (#11769)
f044bc837 Fix bug that `COUNT(DISTINCT)` on StringView panics  (#11768)
0d98b9974 Minor: Add comment explaining rationale for hash check (#11750)
45b40c711 Minor: add "clickbench extended" queries to unit tests (#11763)
6e2ff2955 Derive Debug for logical plan nodes (#11757)
921c3b6b1 Add `TrackedMemoryPool` with better error messages on exhaustion (#11665)
3fe18604e Add null test (#11760)
a4ac0829e Fix documentation warnings, make CsvExecBuilder and Unparsed pub (#11729)
cf98d94c9 Use upstream `DataType::from_str` in arrow-cast (#11254)
1ce546168 Fix `plan_to_sql`: Add wildcard projection to SELECT statement if no projection was set (#11744)
0b8da6d6e Rename RepartitionExec metric `repart_time` to `repartition_time` (#11703)
4884c08ce Use upstream StatisticsConveter (#11479)
9dd2cfc80 Do not push down Sorts if it violates the sort requirements (#11678)
0f554fa12 Minor: Improve documentation for AggregateUDFImpl::state_fields (#11740)
ae2ca6a0e Support  cross-timezone `timestamp` comparison via coercsion (#11711)
fa50636c6 Implement physical plan serialization for parquet Copy plans (#11735)
2887491fc fix: set `null_equals_null` to false when `convert_cross_join_to_inner_join` (#11738)
89677ae66 Check hashes first during probing the aggr hash table (#11718)
abeb8b4f8 Make DefaultSchemaAdapterFactory public (#11709)
6508fa2dc expose some fields on session state (#11716)
929568d60 Use `cargo release` in benchmarks (#11722)
cc6416e74 Minor: Add example for `ScalarUDF::call` (#11727)
8ac50e2db Reduce repetition in try_process_group_by_unnest and try_process_unnest (#11714)
7ca7456d3 Handle nulls in approx_percentile_cont (#11721)
cd786e275 fix: regr_count now returns Uint64 (#11731)
66a85706f Rename `input_type` --> `input_types` on AggregateFunctionExpr / AccumulatorArgs / StateFieldsArgs (#11666)

How are these changes tested?

@andygrove
Copy link
Member Author

Build fails with:

error[E0432]: unresolved imports `datafusion::physical_expr::expressions::Max`, `datafusion::physical_expr::expressions::Min`
  --> core/src/execution/datafusion/planner.rs:36:43
   |
36 |             Literal as DataFusionLiteral, Max, Min, NotExpr,
   |                                           ^^^  ^^^ no `Min` in `expressions`
   |                                           |
   |                                           no `Max` in `expressions`

Looks like we need to update Min and Max as we have done recently for Sum and Count (@huaxingao just fyi)

@huaxingao
Copy link
Contributor

Looks like we need to update Min and Max as we have done recently for Sum and Count

Thanks for pinging me. I will fix this.

@huaxingao
Copy link
Contributor

@andygrove I am not so sure how to push changes to this PR so I have created a new PR instead.

@andygrove andygrove closed this Aug 6, 2024
@andygrove andygrove deleted the bump-df-c6f0d3c branch August 6, 2024 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants