[opt](catalog) merge scan range to avoid too many splits (#38311) #38964
Conversation
PR apache#34032 introduce a new method to get splits batch by batch, but it removed a logic that BE will merge scan ranges to avoid too many scan ranges being scheduled. This PR mainly changes: 1. Add scan range merging logic back. 2. Change the default file split size from 8MB to 64MB, to avoid too many small split.
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
| LOG(INFO) << "Merge " << scan_ranges.size() << " scan ranges to " << merged_ranges.size(); | ||
| } | ||
|
|
||
| protected: |
There was a problem hiding this comment.
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
| protected: |
Additional context
be/src/vec/exec/scan/split_source_connector.h:46: previously declared here
protected:
^|
run buildall |
|
TeamCity be ut coverage result: |
bp #38311