Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](mtmv) Only Generate rewritten plan when generate mv plan for performance #39541

Merged
merged 2 commits into from
Aug 20, 2024
Merged

[improvement](mtmv) Only Generate rewritten plan when generate mv plan for performance #39541

merged 2 commits into from
Aug 20, 2024

Conversation

seawinde
Copy link
Contributor

Proposed changes

Before query rewrite by materialized view, we collecet the table which query used by method org.apache.doris.mtmv.MTMVCache#from.

In MTMVCache#from we calcute the cost of plan which is useless for collecting table.
So add boolean needCost param in method MTMVCache#from to identify that if need cost of plan or not for performance.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38368 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1dda8d3659248eaadd0d2ce86da21484692e4fcf, data reload: false

------ Round 1 ----------------------------------
q1	17858	4409	4306	4306
q2	2043	221	223	221
q3	11747	1029	1091	1029
q4	10530	800	745	745
q5	7793	2842	2773	2773
q6	270	160	160	160
q7	1034	670	658	658
q8	9615	2088	2103	2088
q9	8742	6574	6567	6567
q10	7079	2292	2212	2212
q11	487	275	270	270
q12	429	260	257	257
q13	17782	3017	3002	3002
q14	302	261	254	254
q15	568	528	540	528
q16	522	426	422	422
q17	983	688	659	659
q18	7377	6730	6762	6730
q19	6443	1054	1092	1054
q20	693	360	379	360
q21	3813	3024	3128	3024
q22	1160	1073	1049	1049
Total cold run time: 117270 ms
Total hot run time: 38368 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4525	4301	4289	4289
q2	427	328	319	319
q3	2877	2627	2701	2627
q4	2005	1754	1699	1699
q5	5708	5724	5637	5637
q6	251	155	152	152
q7	2185	1755	1756	1755
q8	3298	3528	3476	3476
q9	8855	8789	8760	8760
q10	3589	3392	3352	3352
q11	633	530	532	530
q12	827	652	670	652
q13	17113	3199	3162	3162
q14	321	293	282	282
q15	558	530	518	518
q16	505	470	453	453
q17	1848	1572	1544	1544
q18	8278	7897	7649	7649
q19	5803	1717	1556	1556
q20	2129	1929	1923	1923
q21	13780	5433	5325	5325
q22	1204	1078	1075	1075
Total cold run time: 86719 ms
Total hot run time: 56735 ms

@doris-robot
Copy link

TPC-H: Total hot run time: 38028 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1dda8d3659248eaadd0d2ce86da21484692e4fcf, data reload: false

------ Round 1 ----------------------------------
q1	17847	4337	4292	4292
q2	2046	215	205	205
q3	11901	972	1091	972
q4	10523	801	848	801
q5	7775	2816	2822	2816
q6	266	156	156	156
q7	1016	644	650	644
q8	9583	2099	2070	2070
q9	8769	6595	6541	6541
q10	7082	2292	2261	2261
q11	482	267	272	267
q12	430	258	254	254
q13	18082	3010	2983	2983
q14	303	254	256	254
q15	549	517	520	517
q16	535	419	405	405
q17	977	716	648	648
q18	7232	6802	6733	6733
q19	6608	1120	1027	1027
q20	694	351	345	345
q21	3801	2806	2890	2806
q22	1132	1053	1031	1031
Total cold run time: 117633 ms
Total hot run time: 38028 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4559	4315	4302	4302
q2	395	300	304	300
q3	2840	2669	2678	2669
q4	2024	1757	1740	1740
q5	5665	5683	5679	5679
q6	274	150	158	150
q7	2204	1812	1815	1812
q8	3339	3552	3459	3459
q9	8771	8765	8743	8743
q10	3637	3366	3267	3267
q11	630	538	525	525
q12	888	682	683	682
q13	16517	3054	3076	3054
q14	342	311	300	300
q15	567	523	527	523
q16	525	449	470	449
q17	1850	1558	1560	1558
q18	8336	7794	7694	7694
q19	7332	1669	1513	1513
q20	2124	1920	1906	1906
q21	14011	5332	5364	5332
q22	1183	1111	1072	1072
Total cold run time: 88013 ms
Total hot run time: 56729 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196921 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1dda8d3659248eaadd0d2ce86da21484692e4fcf, data reload: false

query1	1320	901	884	884
query2	6636	1963	1925	1925
query3	10749	3905	4002	3905
query4	58721	25703	23231	23231
query5	6236	647	674	647
query6	586	210	207	207
query7	6423	335	338	335
query8	549	447	450	447
query9	9183	2543	2517	2517
query10	605	341	330	330
query11	18524	15052	15544	15052
query12	202	162	139	139
query13	1688	498	439	439
query14	12509	7313	7379	7313
query15	340	195	193	193
query16	7452	545	520	520
query17	1475	644	637	637
query18	2088	354	354	354
query19	311	169	171	169
query20	153	138	137	137
query21	261	143	144	143
query22	4723	4691	4641	4641
query23	34906	34214	33555	33555
query24	5717	2989	3003	2989
query25	574	459	441	441
query26	721	186	183	183
query27	1893	311	304	304
query28	3849	2175	2163	2163
query29	671	435	434	434
query30	255	183	187	183
query31	1012	837	806	806
query32	122	78	79	78
query33	549	336	337	336
query34	899	505	493	493
query35	873	791	770	770
query36	1088	956	958	956
query37	154	100	100	100
query38	3951	3885	3833	3833
query39	1525	1462	1463	1462
query40	243	157	154	154
query41	140	165	137	137
query42	135	114	116	114
query43	545	506	505	505
query44	1128	789	788	788
query45	228	194	195	194
query46	1122	800	758	758
query47	1993	1897	1874	1874
query48	424	347	345	345
query49	926	568	583	568
query50	864	468	460	460
query51	6803	6685	6771	6685
query52	128	107	106	106
query53	298	229	226	226
query54	613	501	500	500
query55	89	85	89	85
query56	325	300	304	300
query57	1232	1130	1130	1130
query58	300	293	315	293
query59	3035	2730	2773	2730
query60	353	328	329	328
query61	150	143	143	143
query62	804	699	699	699
query63	254	228	223	223
query64	4312	2340	1851	1851
query65	3232	3220	3205	3205
query66	1062	670	671	670
query67	15395	15214	15049	15049
query68	8445	584	633	584
query69	727	409	340	340
query70	1554	1173	1168	1168
query71	632	319	314	314
query72	6945	2316	2092	2092
query73	2709	361	358	358
query74	9435	8925	8798	8798
query75	5392	2705	2794	2705
query76	5021	1090	1034	1034
query77	811	448	444	444
query78	10229	10300	9532	9532
query79	9763	566	563	563
query80	1404	611	606	606
query81	617	260	262	260
query82	324	155	154	154
query83	341	213	216	213
query84	302	94	100	94
query85	804	357	356	356
query86	371	332	312	312
query87	4405	4141	4309	4141
query88	4388	2497	2487	2487
query89	430	328	325	325
query90	2447	227	230	227
query91	156	125	126	125
query92	84	75	76	75
query93	2828	562	562	562
query94	1323	343	333	333
query95	404	302	345	302
query96	631	293	288	288
query97	3365	3198	3202	3198
query98	250	236	229	229
query99	1648	1331	1331	1331
Total cold run time: 343811 ms
Total hot run time: 196921 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1dda8d3659248eaadd0d2ce86da21484692e4fcf, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.51	0.49	0.51
query6	1.13	0.73	0.74
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.53	0.48	0.49
query10	0.55	0.53	0.55
query11	0.15	0.13	0.12
query12	0.15	0.13	0.14
query13	0.62	0.59	0.58
query14	0.77	0.78	0.78
query15	0.86	0.83	0.82
query16	0.39	0.36	0.39
query17	0.97	0.99	1.05
query18	0.21	0.20	0.20
query19	1.89	1.74	1.73
query20	0.02	0.01	0.01
query21	15.39	0.68	0.66
query22	4.20	7.11	2.70
query23	18.26	1.34	1.25
query24	2.11	0.23	0.22
query25	0.17	0.08	0.09
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.32	1.02	1.01
query29	12.63	3.30	3.30
query30	0.37	0.19	0.21
query31	2.80	0.41	0.41
query32	3.24	0.49	0.49
query33	2.96	3.00	2.92
query34	17.06	4.39	4.39
query35	4.44	4.44	4.45
query36	0.67	0.49	0.50
query37	0.22	0.18	0.17
query38	0.18	0.17	0.17
query39	0.06	0.05	0.06
query40	0.18	0.15	0.15
query41	0.11	0.07	0.07
query42	0.08	0.07	0.06
query43	0.06	0.06	0.07
Total cold run time: 109.72 s
Total hot run time: 31.74 s

@morrySnow
Copy link
Contributor

run beut

@morrySnow
Copy link
Contributor

run cloudut

@morrySnow morrySnow merged commit 3e8c19f into apache:master Aug 20, 2024
29 of 31 checks passed
seawinde added a commit to seawinde/doris that referenced this pull request Aug 21, 2024
@seawinde
[improvement](mtmv) Only Generate rewritten plan when generate mv pl…
89bb669 
…an for performance (apache#39541)

Before query rewrite by materialized view, we collecet the table which
query used by method org.apache.doris.mtmv.MTMVCache#from.

In MTMVCache#from we calcute the cost of plan which is useless for
collecting table.
So add boolean needCost param in method MTMVCache#from to identify
that if need cost of plan or not for performance.
@seawinde seawinde mentioned this pull request Aug 21, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 22, 2024
@seawinde
[fix](mtmv) Fix some pr to 21, prs are (#39041)(#38958)(#39541) (#39678)
021982f 
## Proposed changes

pr: #39041
commitId: 2256298

pr: #38958
commitId: c365cb6

pr: #39541
commitId: 89bb669
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
@seawinde @dataroaring
[improvement](mtmv) Only Generate rewritten plan when generate mv pl…
e6cdbaa 
…an for performance (#39541)

Before query rewrite by materialized view, we collecet the table which
query used by method org.apache.doris.mtmv.MTMVCache#from.

In MTMVCache#from we calcute the cost of plan which is useless for
collecting table.
So add boolean needCost param in method MTMVCache#from to identify
that if need cost of plan or not for performance.
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@924060929 924060929 924060929 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@seawinde @doris-robot @morrySnow @924060929 @yiguolei @dataroaring