[pipelineX](improvement) enable local shuffle by default #28046

Gabriel39 · 2023-12-06T03:38:58Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Gabriel39 · 2023-12-06T03:40:16Z

run buildall

github-actions · 2023-12-06T03:47:04Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-12-06T03:47:37Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei

LGTM

github-actions · 2023-12-06T04:34:42Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-12-06T04:34:44Z

PR approved by anyone and no changes requested.

fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java

doris-robot · 2023-12-06T04:59:53Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 221fd16e89a4897563a6823c7aca0a667f75bcf3, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4657	4431	4428	4428
q2	373	156	156	156
q3	1468	1229	1246	1229
q4	1122	924	890	890
q5	3170	3205	3208	3205
q6	247	132	131	131
q7	1010	497	482	482
q8	2213	2244	2231	2231
q9	6689	6872	6707	6707
q10	3208	3260	3277	3260
q11	334	212	199	199
q12	357	214	209	209
q13	4584	3798	3792	3792
q14	244	216	210	210
q15	567	520	521	520
q16	449	388	383	383
q17	1022	608	540	540
q18	8149	7056	7280	7056
q19	1530	1457	1429	1429
q20	519	344	316	316
q21	3108	2671	2671	2671
q22	352	291	296	291
Total cold run time: 45372 ms
Total hot run time: 40335 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4360	4348	4351	4348
q2	267	169	168	168
q3	3532	3513	3516	3513
q4	2376	2366	2370	2366
q5	5741	5726	5742	5726
q6	242	121	122	121
q7	2389	1871	1864	1864
q8	3527	3507	3525	3507
q9	9077	9039	8959	8959
q10	3916	4007	4017	4007
q11	507	387	396	387
q12	763	594	589	589
q13	4315	3623	3540	3540
q14	281	268	261	261
q15	572	522	518	518
q16	504	468	474	468
q17	1881	1855	1884	1855
q18	8617	8233	8193	8193
q19	1731	1752	1761	1752
q20	2239	1968	1936	1936
q21	6522	6152	6168	6152
q22	484	447	417	417
Total cold run time: 63843 ms
Total hot run time: 60647 ms

doris-robot · 2023-12-06T05:08:59Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.06 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162175063 Bytes

Gabriel39 · 2023-12-06T05:47:07Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h

doris-robot · 2023-12-06T06:14:18Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 59acedf1b874d593434fdbc0500e64a4f658f2f9, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4700	4425	4436	4425
q2	371	150	159	150
q3	1473	1254	1246	1246
q4	1107	938	897	897
q5	3189	3173	3158	3158
q6	248	131	130	130
q7	1008	495	501	495
q8	2198	2210	2204	2204
q9	6666	6741	9891	6741
q10	11974	3254	3295	3254
q11	368	213	195	195
q12	357	214	218	214
q13	14908	3805	3830	3805
q14	250	219	209	209
q15	569	514	517	514
q16	460	392	391	391
q17	1025	597	563	563
q18	7607	7070	7263	7070
q19	1538	1480	1409	1409
q20	528	326	347	326
q21	3075	2686	2680	2680
q22	357	296	306	296
Total cold run time: 63976 ms
Total hot run time: 40372 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4382	4366	4332	4332
q2	273	165	177	165
q3	3547	3518	3526	3518
q4	2376	2372	2373	2372
q5	5747	5777	5733	5733
q6	239	122	122	122
q7	2372	1869	1842	1842
q8	3521	3516	3526	3516
q9	9020	9004	9012	9004
q10	3902	3986	3999	3986
q11	511	381	368	368
q12	774	612	605	605
q13	4333	3562	3606	3562
q14	286	262	259	259
q15	573	513	522	513
q16	515	468	450	450
q17	1868	1852	1850	1850
q18	8703	8258	8349	8258
q19	1725	1750	1741	1741
q20	2274	1947	1937	1937
q21	6504	6231	6128	6128
q22	503	417	412	412
Total cold run time: 63948 ms
Total hot run time: 60673 ms

doris-robot · 2023-12-06T06:25:50Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.8 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17162175106 Bytes

Gabriel39 · 2023-12-06T07:23:03Z

run buildall

doris-robot · 2023-12-06T08:02:22Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.38 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162475932 Bytes

doris-robot · 2023-12-06T08:04:25Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5259537c845ede7c15b2647d74fbe7f8d23e03d0, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4738	4427	4459	4427
q2	374	181	155	155
q3	1459	1271	1209	1209
q4	1124	929	922	922
q5	3164	3179	3164	3164
q6	247	129	128	128
q7	989	502	486	486
q8	2224	2200	2214	2200
q9	6661	6651	6705	6651
q10	3217	3270	3249	3249
q11	320	210	205	205
q12	351	208	204	204
q13	4588	3924	3846	3846
q14	246	211	212	211
q15	563	529	524	524
q16	448	384	391	384
q17	1006	596	554	554
q18	7470	7219	7261	7219
q19	1500	1448	1392	1392
q20	522	325	316	316
q21	3071	2683	2702	2683
q22	352	288	303	288
Total cold run time: 44634 ms
Total hot run time: 40417 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4380	4324	4347	4324
q2	269	159	170	159
q3	3531	3529	3516	3516
q4	2384	2369	2363	2363
q5	5744	5734	5751	5734
q6	236	123	120	120
q7	2369	1849	1852	1849
q8	3523	3522	3526	3522
q9	9075	9028	8986	8986
q10	3910	3975	4003	3975
q11	484	367	389	367
q12	769	596	598	596
q13	4279	3566	3576	3566
q14	288	263	256	256
q15	567	526	521	521
q16	495	467	452	452
q17	1870	1831	1878	1831
q18	8615	8334	8183	8183
q19	1752	1783	1751	1751
q20	2255	1954	1953	1953
q21	6514	6194	6150	6150
q22	511	414	411	411
Total cold run time: 63820 ms
Total hot run time: 60585 ms

yiguolei

LGTM

github-actions · 2023-12-06T08:19:33Z

PR approved by at least one committer and no changes requested.

Gabriel39 added 3 commits December 6, 2023 10:12

[pipelineX](improvement) enable local shuffle by default

131ed67

update

19da931

update

221fd16

github-actions bot added the meta-change label Dec 6, 2023

yiguolei previously approved these changes Dec 6, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 6, 2023

github-actions bot added the reviewed label Dec 6, 2023

morrySnow reviewed Dec 6, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java Show resolved Hide resolved

update

59acedf

Gabriel39 dismissed yiguolei’s stale review via 59acedf December 6, 2023 05:46

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 6, 2023

github-actions bot reviewed Dec 6, 2023

View reviewed changes

update

5259537

yiguolei approved these changes Dec 6, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 6, 2023

BiteTheDDDDt approved these changes Dec 6, 2023

View reviewed changes

Gabriel39 merged commit 2881799 into apache:master Dec 6, 2023

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[pipelineX](improvement) enable local shuffle by default (apache#28046)

e198692

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pipelineX](improvement) enable local shuffle by default #28046

[pipelineX](improvement) enable local shuffle by default #28046

Gabriel39 commented Dec 6, 2023

Gabriel39 commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

yiguolei left a comment

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

doris-robot commented Dec 6, 2023

doris-robot commented Dec 6, 2023

Gabriel39 commented Dec 6, 2023

github-actions bot left a comment

doris-robot commented Dec 6, 2023

doris-robot commented Dec 6, 2023

Gabriel39 commented Dec 6, 2023

doris-robot commented Dec 6, 2023

doris-robot commented Dec 6, 2023

yiguolei left a comment

github-actions bot commented Dec 6, 2023

[pipelineX](improvement) enable local shuffle by default #28046

[pipelineX](improvement) enable local shuffle by default #28046

Conversation

Gabriel39 commented Dec 6, 2023

Proposed changes

Further comments

Gabriel39 commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

yiguolei left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

doris-robot commented Dec 6, 2023

doris-robot commented Dec 6, 2023

Gabriel39 commented Dec 6, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Dec 6, 2023

doris-robot commented Dec 6, 2023

Gabriel39 commented Dec 6, 2023

doris-robot commented Dec 6, 2023

doris-robot commented Dec 6, 2023

yiguolei left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 6, 2023