Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](insert-overwrite) Fix insert overwrite auto detect transaction safe #38103

Merged
merged 4 commits into from
Jul 25, 2024
Merged

[Fix](insert-overwrite) Fix insert overwrite auto detect transaction safe #38103

merged 4 commits into from
Jul 25, 2024

Conversation

zclllyybb
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Before, if insert overwrite auto detect failed because of some transaction conflicts, it will go into an unexpected situation with some of the partition replace success but some retains.
Now it will wholly success, or wholly failed.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39646 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f8da9677620df10da06aad322e5d4c5b45133bcb, data reload: false

------ Round 1 ----------------------------------
q1	17627	4360	4277	4277
q2	2011	188	182	182
q3	10458	1163	1048	1048
q4	10180	794	851	794
q5	7514	2655	2689	2655
q6	220	139	138	138
q7	952	597	584	584
q8	9215	2042	2077	2042
q9	8870	6507	6579	6507
q10	8842	3801	3746	3746
q11	440	233	234	233
q12	392	235	220	220
q13	19018	2991	2974	2974
q14	277	234	237	234
q15	523	484	489	484
q16	501	384	378	378
q17	954	648	727	648
q18	8006	7443	7389	7389
q19	6602	1480	1385	1385
q20	681	316	339	316
q21	4919	3128	3278	3128
q22	330	286	284	284
Total cold run time: 118532 ms
Total hot run time: 39646 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4417	4240	4281	4240
q2	368	260	258	258
q3	2998	2873	2942	2873
q4	1997	1660	1691	1660
q5	5597	5543	5438	5438
q6	225	130	130	130
q7	2193	1904	1846	1846
q8	3259	3440	3398	3398
q9	8782	8894	8815	8815
q10	4154	3748	3843	3748
q11	592	488	504	488
q12	827	674	639	639
q13	17392	3186	3154	3154
q14	328	301	289	289
q15	516	487	487	487
q16	482	440	433	433
q17	1788	1503	1500	1500
q18	8130	7979	7847	7847
q19	1744	1540	1613	1540
q20	2166	1890	1844	1844
q21	5098	4743	4815	4743
q22	603	496	538	496
Total cold run time: 73656 ms
Total hot run time: 55866 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173630 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f8da9677620df10da06aad322e5d4c5b45133bcb, data reload: false

query1	922	371	361	361
query2	6406	1867	1794	1794
query3	6668	212	227	212
query4	28201	17609	17326	17326
query5	3656	470	493	470
query6	250	186	155	155
query7	4578	282	291	282
query8	225	192	202	192
query9	8540	2373	2360	2360
query10	439	290	277	277
query11	10502	10185	10097	10097
query12	117	83	86	83
query13	1665	369	366	366
query14	10164	7697	8063	7697
query15	232	165	171	165
query16	7397	317	321	317
query17	1801	596	550	550
query18	1352	297	283	283
query19	209	162	164	162
query20	94	86	81	81
query21	207	132	126	126
query22	4417	4166	3941	3941
query23	33797	33513	33708	33513
query24	10717	2899	2906	2899
query25	594	434	446	434
query26	696	160	155	155
query27	2303	287	279	279
query28	6139	2065	2041	2041
query29	917	667	643	643
query30	253	163	163	163
query31	1005	765	745	745
query32	95	56	55	55
query33	680	330	396	330
query34	912	498	515	498
query35	689	602	602	602
query36	1111	977	1002	977
query37	140	87	78	78
query38	2945	2865	2819	2819
query39	853	814	811	811
query40	195	121	118	118
query41	44	43	43	43
query42	118	100	109	100
query43	539	491	457	457
query44	1089	726	728	726
query45	192	160	163	160
query46	1095	743	720	720
query47	1870	1811	1753	1753
query48	357	298	294	294
query49	820	410	410	410
query50	788	392	394	392
query51	6811	6862	6784	6784
query52	105	92	94	92
query53	361	282	282	282
query54	862	441	437	437
query55	76	75	72	72
query56	288	265	268	265
query57	1141	1037	1031	1031
query58	247	268	238	238
query59	2912	2576	2778	2576
query60	296	282	269	269
query61	94	91	96	91
query62	794	647	639	639
query63	319	288	291	288
query64	9159	2215	7421	2215
query65	3153	3112	3150	3112
query66	741	330	328	328
query67	15745	15041	14914	14914
query68	5029	536	526	526
query69	689	412	342	342
query70	1189	1049	1062	1049
query71	526	279	273	273
query72	8790	5226	5975	5226
query73	768	332	328	328
query74	6228	5714	5676	5676
query75	4120	2705	2689	2689
query76	3592	938	950	938
query77	701	294	303	294
query78	10657	9067	8968	8968
query79	7314	530	521	521
query80	1398	478	487	478
query81	596	229	218	218
query82	1253	140	134	134
query83	395	213	164	164
query84	273	90	89	89
query85	1446	314	303	303
query86	447	322	323	322
query87	3338	3099	3124	3099
query88	5214	2363	2349	2349
query89	504	390	397	390
query90	1931	192	189	189
query91	134	99	105	99
query92	66	48	48	48
query93	5863	496	491	491
query94	1315	211	212	211
query95	413	320	323	320
query96	617	279	275	275
query97	3179	3011	3019	3011
query98	220	249	197	197
query99	1654	1268	1274	1268
Total cold run time: 291958 ms
Total hot run time: 173630 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f8da9677620df10da06aad322e5d4c5b45133bcb, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.03
query3	0.22	0.05	0.05
query4	1.67	0.09	0.10
query5	0.50	0.50	0.49
query6	1.14	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.55	0.55	0.56
query11	0.15	0.11	0.12
query12	0.14	0.12	0.13
query13	0.59	0.59	0.58
query14	0.75	0.78	0.76
query15	0.84	0.81	0.82
query16	0.35	0.37	0.36
query17	0.98	0.95	0.94
query18	0.22	0.21	0.21
query19	1.80	1.68	1.66
query20	0.01	0.02	0.01
query21	15.38	0.74	0.63
query22	4.02	6.76	2.31
query23	18.29	1.36	1.36
query24	2.16	0.22	0.22
query25	0.16	0.09	0.09
query26	0.30	0.21	0.21
query27	0.45	0.24	0.23
query28	13.26	1.01	0.99
query29	12.60	3.33	3.33
query30	0.25	0.06	0.05
query31	2.89	0.38	0.38
query32	3.30	0.47	0.48
query33	2.90	2.91	2.91
query34	17.19	4.35	4.35
query35	4.42	4.42	4.38
query36	0.66	0.46	0.46
query37	0.19	0.15	0.15
query38	0.16	0.14	0.16
query39	0.05	0.04	0.04
query40	0.15	0.12	0.13
query41	0.09	0.04	0.05
query42	0.05	0.06	0.04
query43	0.05	0.04	0.03
Total cold run time: 109.62 s
Total hot run time: 30.99 s

@HappenLee
Copy link
Contributor

ADD TEST ！

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39540 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ac1d966834c6309d4f260f2f7c390e8cdfd4ab7, data reload: false

------ Round 1 ----------------------------------
q1	17636	4486	4263	4263
q2	2010	206	190	190
q3	10436	1173	1092	1092
q4	10197	793	854	793
q5	7562	2627	2659	2627
q6	218	135	138	135
q7	952	598	604	598
q8	9203	2047	2071	2047
q9	8718	6538	6538	6538
q10	8808	3850	3752	3752
q11	460	233	242	233
q12	445	222	220	220
q13	17772	3024	2993	2993
q14	272	243	250	243
q15	539	489	504	489
q16	493	385	378	378
q17	950	669	706	669
q18	8125	7529	7284	7284
q19	6738	1466	1246	1246
q20	683	330	318	318
q21	4839	3148	3190	3148
q22	350	285	284	284
Total cold run time: 117406 ms
Total hot run time: 39540 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4394	4251	4299	4251
q2	382	265	266	265
q3	2969	2857	2926	2857
q4	2018	1722	1737	1722
q5	5615	5516	5513	5513
q6	230	133	136	133
q7	2193	1892	1874	1874
q8	3275	3400	3441	3400
q9	8853	8772	8927	8772
q10	4083	3839	3803	3803
q11	594	504	533	504
q12	793	633	631	631
q13	16099	3182	3205	3182
q14	310	298	295	295
q15	530	511	489	489
q16	497	429	438	429
q17	1848	1568	1503	1503
q18	8002	7987	7824	7824
q19	1735	1651	1540	1540
q20	2075	1868	1864	1864
q21	5076	4882	4795	4795
q22	595	505	486	486
Total cold run time: 72166 ms
Total hot run time: 56132 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173961 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ac1d966834c6309d4f260f2f7c390e8cdfd4ab7, data reload: false

query1	919	381	361	361
query2	6443	1912	1892	1892
query3	6650	212	214	212
query4	23881	17742	17515	17515
query5	3678	476	473	473
query6	280	164	155	155
query7	4594	286	290	286
query8	234	193	195	193
query9	8515	2430	2422	2422
query10	444	292	290	290
query11	12315	10074	10089	10074
query12	118	82	84	82
query13	1640	370	360	360
query14	10213	7793	7629	7629
query15	252	159	164	159
query16	7744	490	479	479
query17	1572	541	538	538
query18	1637	276	281	276
query19	192	147	153	147
query20	91	81	83	81
query21	203	123	122	122
query22	4336	4045	3902	3902
query23	33993	33865	33590	33590
query24	10066	2957	2869	2869
query25	574	383	409	383
query26	697	154	147	147
query27	2212	276	278	276
query28	5804	2065	2050	2050
query29	891	652	645	645
query30	270	155	151	151
query31	973	758	765	758
query32	94	52	54	52
query33	685	349	344	344
query34	915	495	485	485
query35	853	746	737	737
query36	1150	927	1010	927
query37	151	82	79	79
query38	2903	2859	2861	2859
query39	908	875	842	842
query40	219	123	121	121
query41	45	47	45	45
query42	109	110	114	110
query43	518	479	474	474
query44	1106	710	720	710
query45	194	159	160	159
query46	1081	729	702	702
query47	1843	1797	1771	1771
query48	383	292	290	290
query49	836	401	412	401
query50	765	384	381	381
query51	6753	6676	6664	6664
query52	104	93	93	93
query53	361	322	285	285
query54	870	442	433	433
query55	76	71	72	71
query56	290	284	274	274
query57	1131	1051	1052	1051
query58	249	239	278	239
query59	2916	2776	2554	2554
query60	302	269	266	266
query61	98	99	95	95
query62	779	664	629	629
query63	323	284	291	284
query64	9091	2289	1775	1775
query65	3176	3102	3105	3102
query66	765	336	331	331
query67	15655	15192	14905	14905
query68	4480	531	524	524
query69	474	348	343	343
query70	1223	1137	1113	1113
query71	382	291	291	291
query72	6954	6262	5462	5462
query73	746	322	319	319
query74	6096	5687	5673	5673
query75	3363	2701	2696	2696
query76	2128	1030	911	911
query77	421	325	314	314
query78	9742	9248	9034	9034
query79	2470	525	511	511
query80	2856	497	503	497
query81	600	224	224	224
query82	843	140	141	140
query83	321	173	175	173
query84	270	94	97	94
query85	1851	461	346	346
query86	476	337	312	312
query87	3274	3076	3168	3076
query88	4008	2333	2343	2333
query89	481	376	373	373
query90	1746	189	188	188
query91	129	115	101	101
query92	60	48	50	48
query93	3189	495	489	489
query94	1037	288	288	288
query95	410	324	309	309
query96	599	271	274	271
query97	3206	2987	3079	2987
query98	215	198	204	198
query99	1581	1266	1293	1266
Total cold run time: 274759 ms
Total hot run time: 173961 ms

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

ClickBench: Total hot run time: 30.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ac1d966834c6309d4f260f2f7c390e8cdfd4ab7, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.23	0.06	0.05
query4	1.68	0.07	0.08
query5	0.49	0.49	0.49
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.57	0.51	0.50
query10	0.55	0.54	0.54
query11	0.15	0.11	0.12
query12	0.15	0.13	0.13
query13	0.60	0.60	0.58
query14	0.78	0.77	0.78
query15	0.86	0.81	0.82
query16	0.36	0.36	0.36
query17	1.06	0.96	0.96
query18	0.23	0.22	0.22
query19	1.80	1.77	1.70
query20	0.01	0.00	0.01
query21	15.40	0.77	0.66
query22	4.63	7.20	1.74
query23	18.21	1.42	1.27
query24	2.15	0.21	0.21
query25	0.16	0.09	0.09
query26	0.30	0.22	0.22
query27	0.45	0.23	0.22
query28	13.33	1.02	1.00
query29	12.60	3.32	3.32
query30	0.28	0.06	0.05
query31	2.88	0.39	0.40
query32	3.26	0.47	0.47
query33	2.93	2.90	2.87
query34	17.03	4.42	4.39
query35	4.44	4.45	4.46
query36	0.64	0.46	0.46
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.27 s
Total hot run time: 30.53 s

@doris-robot
Copy link

TPC-H: Total hot run time: 39896 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a75cb1194f51c47e7873567d95952f02f0ff9fe4, data reload: false

------ Round 1 ----------------------------------
q1	18274	4679	4368	4368
q2	2034	191	197	191
q3	10534	1160	1169	1160
q4	10226	887	758	758
q5	7541	2830	2577	2577
q6	221	139	140	139
q7	953	598	604	598
q8	9217	2097	2042	2042
q9	8921	6599	6586	6586
q10	8770	3796	3782	3782
q11	458	240	246	240
q12	394	230	225	225
q13	17746	2980	2975	2975
q14	282	230	235	230
q15	533	479	479	479
q16	488	384	389	384
q17	961	624	731	624
q18	8101	7608	7377	7377
q19	7783	1352	1371	1352
q20	703	320	337	320
q21	4948	3206	3374	3206
q22	347	283	287	283
Total cold run time: 119435 ms
Total hot run time: 39896 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4433	4380	4282	4282
q2	371	266	274	266
q3	3023	2822	2757	2757
q4	1915	1607	1595	1595
q5	5327	5380	5340	5340
q6	221	131	136	131
q7	2109	1780	1758	1758
q8	3203	3375	3304	3304
q9	8500	8443	8525	8443
q10	3922	3710	3712	3710
q11	592	486	501	486
q12	768	607	588	588
q13	16709	3008	2969	2969
q14	307	288	272	272
q15	505	491	470	470
q16	472	418	423	418
q17	1768	1496	1465	1465
q18	7660	7539	7514	7514
q19	1657	1485	1650	1485
q20	2001	1812	1754	1754
q21	4828	4691	4638	4638
q22	591	510	515	510
Total cold run time: 70882 ms
Total hot run time: 54155 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173042 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a75cb1194f51c47e7873567d95952f02f0ff9fe4, data reload: false

query1	921	375	372	372
query2	6447	1905	1907	1905
query3	6645	205	219	205
query4	28740	17562	17287	17287
query5	4234	481	476	476
query6	286	205	167	167
query7	4605	290	275	275
query8	237	201	193	193
query9	8594	2435	2398	2398
query10	438	285	266	266
query11	10945	10003	10106	10003
query12	127	91	83	83
query13	1622	364	362	362
query14	10219	7765	7577	7577
query15	210	167	166	166
query16	7823	491	470	470
query17	1371	532	510	510
query18	1967	274	277	274
query19	198	150	155	150
query20	89	85	81	81
query21	198	125	122	122
query22	4185	4087	3887	3887
query23	33834	33338	33167	33167
query24	11309	2866	2925	2866
query25	630	364	360	360
query26	1458	146	141	141
query27	3006	267	263	263
query28	7754	1979	1969	1969
query29	1001	618	643	618
query30	279	148	148	148
query31	982	752	746	746
query32	98	54	54	54
query33	765	326	317	317
query34	903	470	474	470
query35	855	739	739	739
query36	1100	920	923	920
query37	146	81	76	76
query38	2829	2720	2712	2712
query39	856	818	798	798
query40	272	119	122	119
query41	46	47	46	46
query42	119	100	102	100
query43	517	469	464	464
query44	1195	713	717	713
query45	190	165	160	160
query46	1076	734	742	734
query47	1875	1773	1776	1773
query48	365	283	286	283
query49	1121	412	419	412
query50	797	412	390	390
query51	6669	6771	6654	6654
query52	111	89	93	89
query53	363	286	284	284
query54	949	435	455	435
query55	76	73	75	73
query56	293	260	264	260
query57	1135	1053	1036	1036
query58	241	244	240	240
query59	2819	2662	2811	2662
query60	299	294	310	294
query61	99	94	95	94
query62	841	643	666	643
query63	320	294	289	289
query64	10609	2261	1656	1656
query65	3196	3109	3093	3093
query66	1404	351	349	349
query67	15571	15167	15041	15041
query68	4577	517	524	517
query69	473	339	326	326
query70	1179	1126	1158	1126
query71	403	281	275	275
query72	7065	5764	5897	5764
query73	740	324	323	323
query74	6160	5718	5601	5601
query75	3408	2661	2713	2661
query76	2462	904	989	904
query77	452	312	317	312
query78	9583	9011	9011	9011
query79	2906	530	512	512
query80	2146	479	464	464
query81	573	223	221	221
query82	757	140	134	134
query83	291	172	179	172
query84	277	90	89	89
query85	2113	307	309	307
query86	497	316	319	316
query87	3302	3133	3087	3087
query88	3962	2372	2380	2372
query89	486	379	377	377
query90	1865	194	194	194
query91	139	113	115	113
query92	63	51	52	51
query93	4176	491	490	490
query94	1303	306	304	304
query95	429	323	330	323
query96	608	271	273	271
query97	3231	2978	3033	2978
query98	226	210	201	201
query99	1683	1290	1267	1267
Total cold run time: 288009 ms
Total hot run time: 173042 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a75cb1194f51c47e7873567d95952f02f0ff9fe4, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.09	0.09
query5	0.49	0.50	0.49
query6	1.12	0.74	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.50	0.48
query10	0.55	0.55	0.55
query11	0.16	0.11	0.12
query12	0.15	0.13	0.12
query13	0.60	0.61	0.58
query14	0.77	0.78	0.77
query15	0.86	0.81	0.80
query16	0.36	0.38	0.35
query17	1.05	0.98	0.96
query18	0.21	0.21	0.21
query19	1.80	1.69	1.68
query20	0.02	0.00	0.01
query21	15.39	0.76	0.66
query22	4.05	7.54	1.76
query23	18.28	1.35	1.29
query24	2.11	0.23	0.24
query25	0.16	0.09	0.09
query26	0.29	0.22	0.22
query27	0.46	0.24	0.23
query28	13.29	1.04	1.02
query29	12.69	3.35	3.31
query30	0.25	0.06	0.06
query31	2.87	0.41	0.40
query32	3.25	0.49	0.48
query33	2.85	2.92	2.97
query34	17.05	4.35	4.35
query35	4.51	4.46	4.46
query36	0.65	0.46	0.47
query37	0.19	0.15	0.16
query38	0.17	0.15	0.14
query39	0.04	0.04	0.04
query40	0.15	0.13	0.12
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.03
Total cold run time: 109.65 s
Total hot run time: 30.59 s

@zclllyybb
Copy link
Contributor Author

run buildall

@zclllyybb zclllyybb requested a review from HappenLee July 24, 2024 10:51
@doris-robot
Copy link

TPC-H: Total hot run time: 39702 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 627ab0a8861b2d396567be0b042eafe35b847df1, data reload: false

------ Round 1 ----------------------------------
q1	17594	4312	4216	4216
q2	2011	194	204	194
q3	10440	1256	1102	1102
q4	10183	811	775	775
q5	7535	2678	2679	2678
q6	220	140	139	139
q7	940	604	590	590
q8	9225	2049	2071	2049
q9	8680	6559	6569	6559
q10	8871	3757	3811	3757
q11	457	237	250	237
q12	541	231	233	231
q13	17759	2989	2972	2972
q14	282	231	242	231
q15	525	472	507	472
q16	493	381	381	381
q17	954	558	698	558
q18	8077	7364	7414	7364
q19	6171	1477	1409	1409
q20	666	327	332	327
q21	4965	3182	3336	3182
q22	359	283	279	279
Total cold run time: 116948 ms
Total hot run time: 39702 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4397	4269	4246	4246
q2	377	265	268	265
q3	2972	2822	2928	2822
q4	1994	1706	1681	1681
q5	5702	5533	5508	5508
q6	224	133	135	133
q7	2180	1886	1916	1886
q8	3272	3385	3399	3385
q9	8769	8757	8936	8757
q10	4078	3874	3786	3786
q11	578	509	521	509
q12	824	651	634	634
q13	16330	3235	3213	3213
q14	324	288	282	282
q15	531	480	505	480
q16	515	435	445	435
q17	1809	1524	1508	1508
q18	8024	8144	7797	7797
q19	1766	1569	1587	1569
q20	2134	1897	1855	1855
q21	8752	4943	4837	4837
q22	559	518	522	518
Total cold run time: 76111 ms
Total hot run time: 56106 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174107 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 627ab0a8861b2d396567be0b042eafe35b847df1, data reload: false

query1	912	373	370	370
query2	6448	1886	1822	1822
query3	6651	202	213	202
query4	28105	17638	17370	17370
query5	3658	476	515	476
query6	274	166	164	164
query7	4589	285	283	283
query8	236	195	190	190
query9	8693	2429	2395	2395
query10	439	284	273	273
query11	10802	10073	10085	10073
query12	128	84	85	84
query13	1647	376	376	376
query14	10147	7738	7697	7697
query15	227	172	173	172
query16	7578	518	428	428
query17	1199	534	532	532
query18	1951	274	276	274
query19	187	147	149	147
query20	90	79	81	79
query21	203	123	124	123
query22	4445	4137	4022	4022
query23	34063	34139	33885	33885
query24	11004	2910	2924	2910
query25	575	403	384	384
query26	726	149	149	149
query27	2285	285	287	285
query28	6101	2082	2080	2080
query29	874	656	635	635
query30	250	151	150	150
query31	947	753	774	753
query32	99	55	58	55
query33	647	341	316	316
query34	910	491	492	491
query35	845	772	778	772
query36	1163	979	908	908
query37	141	85	77	77
query38	2951	2888	2845	2845
query39	914	826	824	824
query40	197	122	120	120
query41	45	44	43	43
query42	121	101	98	98
query43	498	444	443	443
query44	1050	720	739	720
query45	206	180	172	172
query46	1093	709	708	708
query47	1857	1787	1777	1777
query48	365	292	281	281
query49	823	409	412	409
query50	763	387	389	387
query51	6797	6669	6678	6669
query52	104	92	91	91
query53	355	286	313	286
query54	885	452	443	443
query55	74	76	73	73
query56	287	263	264	263
query57	1136	1073	1056	1056
query58	244	245	258	245
query59	2755	2574	2653	2574
query60	296	280	283	280
query61	95	91	91	91
query62	775	663	639	639
query63	318	287	285	285
query64	9142	2217	1656	1656
query65	3152	3074	3108	3074
query66	687	338	322	322
query67	15604	15066	14915	14915
query68	4843	546	556	546
query69	612	417	360	360
query70	1160	1158	1089	1089
query71	402	287	276	276
query72	7902	6007	5600	5600
query73	779	325	324	324
query74	6251	5697	5692	5692
query75	3379	2687	2686	2686
query76	2988	911	847	847
query77	611	308	295	295
query78	9976	9374	9023	9023
query79	3207	518	528	518
query80	2129	476	470	470
query81	582	221	218	218
query82	899	141	137	137
query83	311	163	161	161
query84	266	95	82	82
query85	1406	326	308	308
query86	460	305	319	305
query87	3300	3124	3177	3124
query88	4337	2346	2380	2346
query89	491	375	376	375
query90	1806	192	190	190
query91	128	101	99	99
query92	56	49	50	49
query93	4836	516	512	512
query94	1198	300	285	285
query95	402	310	375	310
query96	598	274	274	274
query97	3178	2982	3012	2982
query98	217	194	193	193
query99	1596	1246	1240	1240
Total cold run time: 283134 ms
Total hot run time: 174107 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 627ab0a8861b2d396567be0b042eafe35b847df1, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.04
query4	1.68	0.07	0.07
query5	0.52	0.48	0.48
query6	1.12	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.56	0.50	0.48
query10	0.54	0.54	0.56
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.60	0.58
query14	0.77	0.78	0.77
query15	0.85	0.82	0.81
query16	0.38	0.37	0.36
query17	1.01	0.95	1.02
query18	0.22	0.21	0.22
query19	1.92	1.80	1.81
query20	0.01	0.02	0.02
query21	15.39	0.77	0.64
query22	4.25	6.78	2.18
query23	18.26	1.33	1.28
query24	2.06	0.24	0.25
query25	0.15	0.08	0.08
query26	0.30	0.21	0.21
query27	0.45	0.23	0.23
query28	13.25	1.02	1.00
query29	12.62	3.29	3.28
query30	0.26	0.06	0.06
query31	2.86	0.40	0.39
query32	3.26	0.49	0.48
query33	2.88	2.96	2.95
query34	16.93	4.32	4.45
query35	4.42	4.44	4.38
query36	0.65	0.47	0.48
query37	0.19	0.16	0.16
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.64 s
Total hot run time: 30.96 s

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40143 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 74b2aa1793eec78419807ab426c420f8dc7c7659, data reload: false

------ Round 1 ----------------------------------
q1	18393	4389	4350	4350
q2	2581	196	193	193
q3	11189	1173	1144	1144
q4	10363	817	804	804
q5	7901	2703	2732	2703
q6	220	140	141	140
q7	989	629	616	616
q8	9538	2058	2070	2058
q9	8733	6506	6582	6506
q10	8748	3784	3796	3784
q11	465	251	247	247
q12	401	223	225	223
q13	17755	2955	2993	2955
q14	273	240	232	232
q15	525	478	508	478
q16	497	386	383	383
q17	967	644	643	643
q18	7935	7531	7420	7420
q19	6944	1395	1429	1395
q20	707	327	313	313
q21	5023	3278	3272	3272
q22	356	306	284	284
Total cold run time: 120503 ms
Total hot run time: 40143 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4343	4264	4291	4264
q2	387	275	260	260
q3	3024	2738	2796	2738
q4	1880	1623	1589	1589
q5	5335	5331	5300	5300
q6	217	131	133	131
q7	2163	1729	1708	1708
q8	3192	3352	3310	3310
q9	8461	8455	8417	8417
q10	3904	3712	3795	3712
q11	588	496	486	486
q12	758	633	610	610
q13	16764	3013	3042	3013
q14	313	300	276	276
q15	519	480	481	480
q16	475	419	414	414
q17	1779	1507	1469	1469
q18	7710	7595	7463	7463
q19	1659	1523	1505	1505
q20	1991	1800	1830	1800
q21	4950	4703	4805	4703
q22	563	498	519	498
Total cold run time: 70975 ms
Total hot run time: 54146 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173147 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 74b2aa1793eec78419807ab426c420f8dc7c7659, data reload: false

query1	915	374	374	374
query2	6441	1896	1807	1807
query3	6648	203	212	203
query4	22893	17535	17470	17470
query5	4186	460	477	460
query6	277	183	160	160
query7	4587	292	285	285
query8	243	200	205	200
query9	8469	2422	2400	2400
query10	445	306	276	276
query11	10791	10162	10135	10135
query12	136	88	85	85
query13	1659	396	370	370
query14	10169	8016	7644	7644
query15	219	166	164	164
query16	7414	472	443	443
query17	1573	536	521	521
query18	1435	272	271	271
query19	195	147	147	147
query20	86	85	80	80
query21	210	124	145	124
query22	4483	3932	3959	3932
query23	33795	33095	33263	33095
query24	11844	2965	2855	2855
query25	660	367	370	367
query26	1725	150	146	146
query27	2737	266	293	266
query28	7490	2014	1999	1999
query29	1013	619	608	608
query30	286	148	146	146
query31	1024	726	737	726
query32	94	52	52	52
query33	772	320	321	320
query34	908	482	478	478
query35	882	738	725	725
query36	1110	948	963	948
query37	156	85	79	79
query38	2837	2754	2774	2754
query39	890	817	808	808
query40	275	121	120	120
query41	48	46	44	44
query42	114	99	104	99
query43	497	482	456	456
query44	1226	720	732	720
query45	202	176	173	173
query46	1090	731	687	687
query47	1839	1791	1763	1763
query48	381	291	281	281
query49	1166	408	414	408
query50	806	396	391	391
query51	6794	6733	6684	6684
query52	103	87	93	87
query53	354	290	289	289
query54	992	455	442	442
query55	73	72	74	72
query56	296	264	273	264
query57	1153	1051	1068	1051
query58	257	255	257	255
query59	2913	2698	2645	2645
query60	289	278	262	262
query61	97	101	127	101
query62	815	659	659	659
query63	320	303	282	282
query64	10608	2218	1670	1670
query65	3249	3142	3087	3087
query66	1308	341	328	328
query67	15373	15106	14839	14839
query68	4650	540	561	540
query69	538	477	356	356
query70	1210	1155	1119	1119
query71	397	276	271	271
query72	7242	5750	5806	5750
query73	776	330	327	327
query74	6145	5653	5716	5653
query75	3396	2676	2714	2676
query76	2599	881	872	872
query77	492	299	297	297
query78	11756	9314	8953	8953
query79	4254	519	529	519
query80	1127	479	472	472
query81	569	219	221	219
query82	695	133	129	129
query83	196	173	168	168
query84	279	86	84	84
query85	706	310	314	310
query86	468	338	330	330
query87	3288	3076	3081	3076
query88	4711	2385	2383	2383
query89	505	395	389	389
query90	1953	189	194	189
query91	140	112	177	112
query92	69	47	49	47
query93	4280	515	507	507
query94	1132	290	288	288
query95	400	321	307	307
query96	610	269	278	269
query97	3198	3014	2997	2997
query98	232	202	195	195
query99	1622	1283	1261	1261
Total cold run time: 283970 ms
Total hot run time: 173147 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74b2aa1793eec78419807ab426c420f8dc7c7659, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.22	0.05	0.06
query4	1.68	0.08	0.07
query5	0.50	0.50	0.49
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.47	0.50
query10	0.54	0.53	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.76	0.77	0.78
query15	0.87	0.82	0.82
query16	0.35	0.38	0.37
query17	0.99	1.04	0.98
query18	0.22	0.22	0.21
query19	1.85	1.85	1.81
query20	0.02	0.01	0.01
query21	15.40	0.77	0.65
query22	3.83	6.94	2.15
query23	18.30	1.39	1.24
query24	2.11	0.22	0.21
query25	0.16	0.08	0.08
query26	0.30	0.20	0.21
query27	0.45	0.24	0.23
query28	13.24	1.02	1.00
query29	12.59	3.36	3.30
query30	0.25	0.06	0.05
query31	2.87	0.38	0.39
query32	3.28	0.48	0.47
query33	2.91	2.92	2.90
query34	16.99	4.32	4.27
query35	4.42	4.41	4.42
query36	0.66	0.48	0.48
query37	0.20	0.16	0.16
query38	0.16	0.16	0.16
query39	0.04	0.04	0.04
query40	0.15	0.13	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.06
query43	0.04	0.04	0.04
Total cold run time: 109.28 s
Total hot run time: 30.87 s

HappenLee
HappenLee approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit 1c18b4c into apache:master Jul 25, 2024
28 of 30 checks passed
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Jul 28, 2024
@zclllyybb
[Fix](insert-overwrite) Fix insert overwrite auto detect transaction …
c838799 
…safe (apache#38103)

Before, if insert overwrite auto detect failed because of some
transaction conflicts, it will go into an unexpected situation with some
of the partition replace success but some retains.
Now it will wholly success, or wholly failed.
@zclllyybb zclllyybb mentioned this pull request Jul 28, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 29, 2024
@zclllyybb
[branch-2.1](insert-overwrite) Fix insert overwrite auto detect trans…
dab0138 
…action safe (#38103) (#38442)

pick #38103
dataroaring pushed a commit that referenced this pull request Jul 31, 2024
@zclllyybb @dataroaring
[Fix](insert-overwrite) Fix insert overwrite auto detect transaction …
8abf99e 
…safe (#38103)

Before, if insert overwrite auto detect failed because of some
transaction conflicts, it will go into an unexpected situation with some
of the partition replace success but some retains.
Now it will wholly success, or wholly failed.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@zclllyybb zclllyybb deleted the fix_iot branch August 30, 2024 02:27
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HappenLee HappenLee HappenLee approved these changes

@Gabriel39 Gabriel39 Gabriel39 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zclllyybb @doris-robot @HappenLee @Gabriel39 @wm1581066 @dataroaring