Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve](mow) refactor mow update lock for compaction to reduce txn conflict #48024

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Feb 18, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Feb 18, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31593 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 51771b5b754068068b7d91e49dff1e19db99068e, data reload: false

------ Round 1 ----------------------------------
q1	17620	5239	5104	5104
q2	2057	308	186	186
q3	10470	1268	788	788
q4	10300	1022	552	552
q5	8354	2379	2386	2379
q6	195	166	133	133
q7	925	771	595	595
q8	9323	1318	1057	1057
q9	4977	4522	4847	4522
q10	6821	2316	1911	1911
q11	486	277	268	268
q12	352	367	227	227
q13	17759	3729	3058	3058
q14	231	220	208	208
q15	521	472	472	472
q16	625	612	570	570
q17	578	880	341	341
q18	6707	6183	6407	6183
q19	2350	968	572	572
q20	306	320	185	185
q21	2987	2085	1974	1974
q22	364	340	308	308
Total cold run time: 104308 ms
Total hot run time: 31593 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5329	5174	5189	5174
q2	242	328	232	232
q3	2188	2716	2298	2298
q4	1478	1816	1374	1374
q5	4270	4203	4231	4203
q6	204	161	123	123
q7	1885	1916	1749	1749
q8	2669	2658	2587	2587
q9	7202	7105	7155	7105
q10	3017	3233	2806	2806
q11	574	498	481	481
q12	690	801	592	592
q13	3386	3970	3330	3330
q14	271	297	276	276
q15	522	464	464	464
q16	629	692	637	637
q17	1139	1622	1350	1350
q18	7567	7525	7394	7394
q19	830	836	973	836
q20	1974	2033	1884	1884
q21	5536	5066	4679	4679
q22	613	598	586	586
Total cold run time: 52215 ms
Total hot run time: 50160 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189747 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 51771b5b754068068b7d91e49dff1e19db99068e, data reload: false

query1	1321	991	921	921
query2	6209	1897	1874	1874
query3	10956	4341	4495	4341
query4	53417	25417	23037	23037
query5	5284	552	485	485
query6	367	192	211	192
query7	4962	520	296	296
query8	329	243	244	243
query9	6103	2507	2513	2507
query10	420	304	248	248
query11	15650	15138	14814	14814
query12	154	104	105	104
query13	1096	506	408	408
query14	10174	6643	6399	6399
query15	200	195	178	178
query16	7034	644	431	431
query17	1059	716	560	560
query18	1516	401	305	305
query19	214	197	172	172
query20	128	134	129	129
query21	220	131	106	106
query22	4369	4540	4381	4381
query23	34048	33378	33312	33312
query24	6008	2437	2457	2437
query25	484	517	418	418
query26	669	289	163	163
query27	1973	487	340	340
query28	2946	2463	2470	2463
query29	560	555	425	425
query30	212	185	151	151
query31	899	849	821	821
query32	79	65	68	65
query33	447	349	303	303
query34	763	848	506	506
query35	830	833	746	746
query36	964	992	897	897
query37	115	106	73	73
query38	4295	4381	4275	4275
query39	1520	1442	1457	1442
query40	210	119	101	101
query41	52	52	51	51
query42	124	110	108	108
query43	517	526	478	478
query44	1351	843	839	839
query45	178	177	163	163
query46	892	1076	664	664
query47	1852	1883	1776	1776
query48	394	412	312	312
query49	702	525	407	407
query50	721	766	427	427
query51	4325	4286	4241	4241
query52	106	104	106	104
query53	233	253	195	195
query54	478	484	422	422
query55	78	82	79	79
query56	267	280	246	246
query57	1157	1159	1117	1117
query58	243	257	241	241
query59	2834	2941	2708	2708
query60	299	286	260	260
query61	120	133	121	121
query62	746	743	670	670
query63	238	208	190	190
query64	1420	1019	725	725
query65	3245	3160	3123	3123
query66	786	405	303	303
query67	16011	15530	15430	15430
query68	5747	778	515	515
query69	531	320	272	272
query70	1186	1124	1100	1100
query71	435	287	266	266
query72	5987	3620	3779	3620
query73	1025	752	356	356
query74	9026	8973	8904	8904
query75	3194	3145	2654	2654
query76	3992	1178	745	745
query77	591	360	278	278
query78	9980	10157	9360	9360
query79	2142	814	599	599
query80	610	517	447	447
query81	506	281	238	238
query82	254	126	92	92
query83	171	177	154	154
query84	294	94	75	75
query85	821	349	299	299
query86	380	320	297	297
query87	4356	4648	4553	4553
query88	3506	2232	2205	2205
query89	392	318	284	284
query90	1721	196	192	192
query91	137	145	108	108
query92	70	60	58	58
query93	1913	1003	581	581
query94	686	409	284	284
query95	344	271	260	260
query96	486	572	267	267
query97	2857	2846	2743	2743
query98	238	197	200	197
query99	1296	1378	1275	1275
Total cold run time: 292447 ms
Total hot run time: 189747 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 51771b5b754068068b7d91e49dff1e19db99068e, data reload: false

query1	0.03	0.03	0.03
query2	0.10	0.05	0.04
query3	0.27	0.06	0.06
query4	1.59	0.08	0.08
query5	0.42	0.40	0.40
query6	1.17	0.65	0.65
query7	0.02	0.02	0.01
query8	0.07	0.04	0.04
query9	0.61	0.51	0.51
query10	0.57	0.58	0.57
query11	0.24	0.12	0.12
query12	0.25	0.13	0.12
query13	0.63	0.61	0.62
query14	2.67	2.69	2.74
query15	0.98	0.86	0.86
query16	0.37	0.38	0.37
query17	1.04	1.04	1.02
query18	0.18	0.18	0.18
query19	1.99	1.81	2.03
query20	0.01	0.01	0.02
query21	15.35	0.96	0.65
query22	0.91	1.04	0.79
query23	14.71	1.53	0.72
query24	7.86	0.87	0.40
query25	0.16	0.09	0.09
query26	0.61	0.23	0.18
query27	0.09	0.09	0.09
query28	11.10	1.17	0.55
query29	12.57	4.01	3.31
query30	0.27	0.07	0.05
query31	2.82	0.61	0.42
query32	3.22	0.59	0.50
query33	2.98	3.05	3.13
query34	16.49	5.17	4.45
query35	4.48	4.45	4.47
query36	0.63	0.50	0.48
query37	0.20	0.17	0.17
query38	0.17	0.16	0.15
query39	0.05	0.05	0.04
query40	0.20	0.16	0.16
query41	0.10	0.05	0.05
query42	0.06	0.06	0.05
query43	0.06	0.04	0.04
Total cold run time: 108.3 s
Total hot run time: 31.02 s

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Feb 19, 2025

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.46% (1067/1294)
Line Coverage: 65.88% (17802/27022)
Region Coverage: 65.49% (8778/13404)
Branch Coverage: 55.32% (4723/8538)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6cbdf083a0365a449166dc3b3a5eab5a6940ad5b_6cbdf083a0365a449166dc3b3a5eab5a6940ad5b_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 31567 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6cbdf083a0365a449166dc3b3a5eab5a6940ad5b, data reload: false

------ Round 1 ----------------------------------
q1	17673	5384	5117	5117
q2	2070	309	177	177
q3	10515	1341	755	755
q4	10235	1034	543	543
q5	7670	2489	2304	2304
q6	199	171	137	137
q7	923	763	612	612
q8	9315	1351	1147	1147
q9	4869	4704	4610	4610
q10	6834	2333	1883	1883
q11	479	269	259	259
q12	343	357	210	210
q13	17802	3648	3080	3080
q14	222	221	201	201
q15	509	475	449	449
q16	621	605	578	578
q17	583	879	338	338
q18	6845	6364	6223	6223
q19	1981	963	550	550
q20	306	310	191	191
q21	2747	2084	1914	1914
q22	366	332	289	289
Total cold run time: 103107 ms
Total hot run time: 31567 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5168	5098	5099	5098
q2	236	329	225	225
q3	2170	2610	2316	2316
q4	1441	1824	1394	1394
q5	4230	4302	4110	4110
q6	211	160	123	123
q7	1844	1852	1758	1758
q8	2645	2977	2609	2609
q9	7270	7051	7089	7051
q10	2999	3192	2738	2738
q11	578	520	493	493
q12	667	755	622	622
q13	3501	3957	3221	3221
q14	274	287	280	280
q15	491	452	460	452
q16	629	675	639	639
q17	1149	1621	1353	1353
q18	7583	7364	7377	7364
q19	766	820	1036	820
q20	1980	1993	1879	1879
q21	5253	4918	4775	4775
q22	629	570	543	543
Total cold run time: 51714 ms
Total hot run time: 49863 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183435 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6cbdf083a0365a449166dc3b3a5eab5a6940ad5b, data reload: false

query1	972	386	390	386
query2	6548	1827	1862	1827
query3	6796	211	213	211
query4	26296	23645	23538	23538
query5	4492	684	487	487
query6	318	202	199	199
query7	4614	505	306	306
query8	282	241	221	221
query9	8632	2538	2553	2538
query10	447	307	246	246
query11	15313	15040	14933	14933
query12	151	105	103	103
query13	1645	522	385	385
query14	9191	6074	6594	6074
query15	220	206	178	178
query16	7387	639	461	461
query17	1208	718	569	569
query18	1962	398	311	311
query19	197	193	164	164
query20	125	119	118	118
query21	210	123	105	105
query22	4167	4218	4208	4208
query23	33913	32961	33014	32961
query24	7697	2359	2393	2359
query25	533	454	378	378
query26	1231	269	157	157
query27	2281	480	347	347
query28	4036	2442	2399	2399
query29	781	526	459	459
query30	232	185	157	157
query31	944	857	776	776
query32	79	64	68	64
query33	574	364	316	316
query34	767	844	523	523
query35	802	812	724	724
query36	966	958	908	908
query37	123	102	71	71
query38	4192	4074	4064	4064
query39	1427	1393	1401	1393
query40	208	114	104	104
query41	57	59	53	53
query42	127	99	102	99
query43	481	494	459	459
query44	1265	801	798	798
query45	182	163	170	163
query46	865	1033	668	668
query47	1756	1810	1732	1732
query48	387	434	301	301
query49	785	510	435	435
query50	672	741	409	409
query51	4189	4134	4106	4106
query52	105	107	91	91
query53	227	255	188	188
query54	491	487	414	414
query55	83	78	75	75
query56	272	275	256	256
query57	1124	1142	1059	1059
query58	247	233	242	233
query59	2625	2612	2461	2461
query60	285	270	267	267
query61	124	120	147	120
query62	775	743	664	664
query63	242	201	187	187
query64	4481	980	697	697
query65	3237	3128	3264	3128
query66	1144	396	304	304
query67	15697	15402	15281	15281
query68	5628	781	532	532
query69	468	293	271	271
query70	1201	1129	1124	1124
query71	401	305	269	269
query72	5722	3533	3741	3533
query73	744	737	359	359
query74	9212	9333	8977	8977
query75	3140	3178	2686	2686
query76	3159	1156	732	732
query77	475	363	285	285
query78	9901	10492	9287	9287
query79	1421	876	586	586
query80	660	532	432	432
query81	477	282	238	238
query82	192	129	97	97
query83	167	185	157	157
query84	236	92	73	73
query85	745	340	308	308
query86	315	306	276	276
query87	4576	4375	4437	4375
query88	2919	2220	2237	2220
query89	376	320	294	294
query90	1770	192	188	188
query91	131	142	111	111
query92	62	60	58	58
query93	1139	1010	591	591
query94	592	414	299	299
query95	352	263	255	255
query96	504	523	271	271
query97	2748	2846	2735	2735
query98	222	205	197	197
query99	1345	1394	1243	1243
Total cold run time: 264884 ms
Total hot run time: 183435 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6cbdf083a0365a449166dc3b3a5eab5a6940ad5b, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.46	0.42	0.40
query6	1.17	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.61	0.51	0.53
query10	0.58	0.57	0.57
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.61	0.61
query14	2.70	2.70	2.72
query15	0.91	0.86	0.85
query16	0.38	0.38	0.36
query17	1.02	1.04	1.05
query18	0.21	0.19	0.19
query19	1.94	2.00	1.83
query20	0.01	0.01	0.02
query21	15.38	0.90	0.53
query22	0.74	1.28	0.75
query23	14.76	1.39	0.63
query24	9.80	1.51	0.68
query25	0.50	0.22	0.13
query26	0.86	0.18	0.14
query27	0.05	0.05	0.05
query28	6.42	0.80	0.42
query29	12.55	3.98	3.26
query30	0.25	0.09	0.06
query31	2.83	0.58	0.39
query32	3.23	0.55	0.47
query33	2.94	3.09	3.08
query34	15.62	5.07	4.42
query35	4.50	4.49	4.53
query36	0.66	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.48 s
Total hot run time: 30.49 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants