Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](statistics)Return -1 when external table row count is unknown. #38990

Merged
merged 1 commit into from
Aug 8, 2024
Merged

[improvement](statistics)Return -1 when external table row count is unknown. #38990

merged 1 commit into from
Aug 8, 2024

Conversation

Jibing-Li
Copy link
Contributor

Return -1 when external table row count is unknown.
Don't cache any row count value when loading row count for external table get exception.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review August 7, 2024 02:02
@Jibing-Li
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the doing label Aug 7, 2024
@Jibing-Li
Copy link
Contributor Author

run buildall

@Jibing-Li
Copy link
Contributor Author

run performance

@Jibing-Li
Copy link
Contributor Author

run buildall

@morrySnow morrySnow removed the doing label Aug 7, 2024
@github-actions github-actions bot added the doing label Aug 7, 2024
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41066 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5ecce2c972dccaffdb84dbb8e00f53bd1654a522, data reload: false

------ Round 1 ----------------------------------
q1	17714	4070	4040	4040
q2	2022	202	192	192
q3	10476	1248	1341	1248
q4	10153	813	842	813
q5	7610	2935	2949	2935
q6	219	135	134	134
q7	1035	608	607	607
q8	9429	1788	1954	1788
q9	8404	6546	6536	6536
q10	8753	3829	3800	3800
q11	437	248	246	246
q12	412	233	224	224
q13	17752	2955	2918	2918
q14	268	244	243	243
q15	521	476	487	476
q16	517	387	389	387
q17	961	927	917	917
q18	8089	7331	7199	7199
q19	3058	1070	1210	1070
q20	550	327	344	327
q21	5239	4688	4700	4688
q22	347	278	283	278
Total cold run time: 113966 ms
Total hot run time: 41066 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4091	4032	3993	3993
q2	324	223	220	220
q3	2991	2964	3128	2964
q4	2002	1980	1946	1946
q5	5615	5444	5433	5433
q6	221	130	129	129
q7	2122	1803	1850	1803
q8	3287	3358	3336	3336
q9	8587	8601	8639	8601
q10	3971	4043	3903	3903
q11	568	466	466	466
q12	804	561	609	561
q13	16389	3104	3122	3104
q14	295	287	283	283
q15	542	494	478	478
q16	463	415	429	415
q17	1775	1706	1745	1706
q18	8207	7678	7671	7671
q19	1782	1702	1707	1702
q20	2129	1846	1846	1846
q21	5772	5443	5216	5216
q22	528	466	477	466
Total cold run time: 72465 ms
Total hot run time: 56242 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169296 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5ecce2c972dccaffdb84dbb8e00f53bd1654a522, data reload: false

query1	926	365	362	362
query2	6455	1721	1651	1651
query3	6665	209	214	209
query4	20373	17421	17426	17421
query5	3676	517	513	513
query6	275	161	164	161
query7	4591	295	293	293
query8	259	195	201	195
query9	8510	2344	2326	2326
query10	456	297	272	272
query11	10792	10099	10073	10073
query12	120	91	87	87
query13	1639	381	381	381
query14	9998	7688	6947	6947
query15	206	163	173	163
query16	6931	463	460	460
query17	957	585	560	560
query18	1841	288	285	285
query19	190	151	150	150
query20	93	86	89	86
query21	207	148	96	96
query22	4365	4126	4050	4050
query23	33993	33428	33209	33209
query24	9222	3192	3094	3094
query25	650	401	417	401
query26	1470	152	152	152
query27	2823	278	296	278
query28	7337	2000	1985	1985
query29	1081	440	440	440
query30	246	161	156	156
query31	943	786	776	776
query32	105	57	56	56
query33	679	306	322	306
query34	915	494	504	494
query35	899	761	774	761
query36	1022	895	874	874
query37	178	81	80	80
query38	2903	2827	2851	2827
query39	884	819	809	809
query40	250	110	110	110
query41	47	48	48	48
query42	115	99	100	99
query43	476	422	430	422
query44	1124	722	752	722
query45	206	175	174	174
query46	1069	790	806	790
query47	1848	1750	1763	1750
query48	365	292	286	286
query49	944	418	417	417
query50	892	425	445	425
query51	6711	6681	6746	6681
query52	100	91	91	91
query53	252	182	175	175
query54	610	449	448	448
query55	75	78	76	76
query56	266	247	254	247
query57	1123	1066	1040	1040
query58	274	259	287	259
query59	2553	2402	2346	2346
query60	293	280	277	277
query61	99	95	91	91
query62	913	650	662	650
query63	213	183	180	180
query64	5539	1888	1885	1885
query65	3145	3075	3099	3075
query66	1070	329	327	327
query67	15213	14791	14770	14770
query68	6002	575	588	575
query69	710	367	316	316
query70	1168	1080	1011	1011
query71	434	272	267	267
query72	7526	2662	2520	2520
query73	917	326	330	326
query74	6119	5719	5660	5660
query75	3528	2744	2691	2691
query76	3350	1267	1223	1223
query77	653	312	307	307
query78	9509	8838	8793	8793
query79	1373	540	539	539
query80	842	512	509	509
query81	565	228	228	228
query82	835	138	131	131
query83	214	170	166	166
query84	272	76	78	76
query85	1464	371	295	295
query86	470	312	305	305
query87	3305	3127	3106	3106
query88	3693	2380	2384	2380
query89	381	284	288	284
query90	1774	194	194	194
query91	129	97	99	97
query92	59	51	55	51
query93	2027	615	625	615
query94	777	302	282	282
query95	377	271	276	271
query96	615	283	280	280
query97	3266	3083	3034	3034
query98	228	202	204	202
query99	1608	1297	1262	1262
Total cold run time: 264850 ms
Total hot run time: 169296 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5ecce2c972dccaffdb84dbb8e00f53bd1654a522, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.48	0.48	0.47
query6	1.14	0.71	0.71
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.56	0.52	0.52
query10	0.57	0.57	0.56
query11	0.15	0.11	0.11
query12	0.15	0.13	0.12
query13	0.62	0.60	0.60
query14	0.76	0.79	0.79
query15	0.90	0.87	0.85
query16	0.36	0.37	0.35
query17	1.00	0.99	0.98
query18	0.21	0.21	0.20
query19	1.78	1.72	1.72
query20	0.00	0.03	0.01
query21	15.39	0.75	0.64
query22	3.73	7.06	1.93
query23	18.08	1.26	1.29
query24	2.27	0.22	0.22
query25	0.19	0.08	0.08
query26	0.31	0.22	0.22
query27	0.45	0.23	0.23
query28	13.17	1.01	0.97
query29	12.56	3.27	3.28
query30	0.25	0.06	0.05
query31	2.87	0.42	0.40
query32	3.24	0.49	0.48
query33	2.94	2.98	2.92
query34	15.43	4.26	4.27
query35	4.27	4.27	4.30
query36	0.66	0.47	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.17	0.13	0.13
query41	0.11	0.04	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.05
Total cold run time: 107.36 s
Total hot run time: 30.49 s

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39185 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3efca2d39730334ff210701fc8875a0ea35bf88a, data reload: false

------ Round 1 ----------------------------------
q1	17933	4741	4295	4295
q2	2012	177	174	174
q3	10550	1154	1125	1125
q4	10177	735	626	626
q5	7511	2548	2558	2548
q6	226	138	135	135
q7	971	588	603	588
q8	9210	1910	1935	1910
q9	8593	6651	6598	6598
q10	7090	2190	2207	2190
q11	461	230	242	230
q12	402	215	217	215
q13	18765	2985	3007	2985
q14	271	243	237	237
q15	520	491	497	491
q16	509	391	380	380
q17	993	686	696	686
q18	8191	7399	7465	7399
q19	2977	1000	1059	1000
q20	645	310	311	310
q21	5749	4083	4348	4083
q22	1135	1009	980	980
Total cold run time: 114891 ms
Total hot run time: 39185 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4530	4276	4295	4276
q2	372	282	261	261
q3	2853	2611	2636	2611
q4	1894	1582	1637	1582
q5	5275	5277	5291	5277
q6	223	132	132	132
q7	2041	1679	1633	1633
q8	3173	3387	3343	3343
q9	8409	8390	8379	8379
q10	3389	3153	3199	3153
q11	602	496	491	491
q12	784	578	607	578
q13	17520	3007	2972	2972
q14	307	280	270	270
q15	523	490	478	478
q16	474	406	409	406
q17	1803	1482	1460	1460
q18	7638	7589	7227	7227
q19	1664	1507	1537	1507
q20	1991	1804	1788	1788
q21	5219	5116	5050	5050
q22	1098	1004	1008	1004
Total cold run time: 71782 ms
Total hot run time: 53878 ms

@Jibing-Li
Copy link
Contributor Author

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 39836 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3efca2d39730334ff210701fc8875a0ea35bf88a, data reload: false

------ Round 1 ----------------------------------
q1	17651	4390	4380	4380
q2	2020	183	189	183
q3	10481	1223	998	998
q4	10148	747	798	747
q5	7524	2639	2540	2540
q6	224	143	139	139
q7	986	602	600	600
q8	9414	1963	1949	1949
q9	10348	6603	6577	6577
q10	7073	2174	2248	2174
q11	462	244	244	244
q12	465	223	217	217
q13	17995	3000	2991	2991
q14	273	249	234	234
q15	530	494	506	494
q16	541	375	389	375
q17	1004	715	729	715
q18	8086	7522	7397	7397
q19	1375	1020	1015	1015
q20	657	322	330	322
q21	5336	4609	4566	4566
q22	1115	1028	979	979
Total cold run time: 113708 ms
Total hot run time: 39836 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4344	4281	4278	4278
q2	381	272	262	262
q3	2869	2581	2673	2581
q4	1863	1591	1608	1591
q5	5323	5269	5244	5244
q6	218	131	135	131
q7	2045	1695	1666	1666
q8	3167	3360	3322	3322
q9	8474	8373	8414	8373
q10	3370	3120	3122	3120
q11	591	492	478	478
q12	775	598	572	572
q13	16394	2977	2964	2964
q14	301	277	271	271
q15	524	478	476	476
q16	477	412	413	412
q17	1781	1501	1464	1464
q18	7516	7434	7417	7417
q19	1670	1508	1485	1485
q20	1988	1763	1764	1763
q21	5316	5022	5058	5022
q22	1108	988	980	980
Total cold run time: 70495 ms
Total hot run time: 53872 ms

@Jibing-Li
Copy link
Contributor Author

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 39556 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3efca2d39730334ff210701fc8875a0ea35bf88a, data reload: false

------ Round 1 ----------------------------------
q1	18051	4474	4435	4435
q2	2458	177	183	177
q3	11508	1157	1097	1097
q4	10451	793	715	715
q5	7696	2571	2485	2485
q6	226	142	139	139
q7	1007	605	604	604
q8	9581	1902	1915	1902
q9	8538	6570	6549	6549
q10	7063	2151	2148	2148
q11	451	248	244	244
q12	389	222	219	219
q13	18816	2959	2979	2959
q14	284	239	227	227
q15	531	471	498	471
q16	503	393	379	379
q17	957	619	711	619
q18	7971	7364	7527	7364
q19	5891	1080	1052	1052
q20	682	324	327	324
q21	5282	4466	4440	4440
q22	1106	1007	1007	1007
Total cold run time: 119442 ms
Total hot run time: 39556 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4518	4267	4263	4263
q2	395	271	262	262
q3	2820	2610	2617	2610
q4	1892	1612	1635	1612
q5	5280	5209	5230	5209
q6	228	134	134	134
q7	2023	1644	1638	1638
q8	3159	3336	3321	3321
q9	8323	8335	8327	8327
q10	3399	3127	3158	3127
q11	595	502	507	502
q12	774	630	611	611
q13	16958	2960	2993	2960
q14	291	285	300	285
q15	517	485	487	485
q16	487	420	416	416
q17	1744	1497	1468	1468
q18	7523	7617	7352	7352
q19	2148	1660	1617	1617
q20	2033	1775	1760	1760
q21	5145	5064	5038	5038
q22	1081	994	1032	994
Total cold run time: 71333 ms
Total hot run time: 53991 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202111 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3efca2d39730334ff210701fc8875a0ea35bf88a, data reload: false

query1	950	414	393	393
query2	6446	1988	1865	1865
query3	6650	212	218	212
query4	34155	23437	23134	23134
query5	4180	487	492	487
query6	292	178	183	178
query7	4582	298	291	291
query8	245	210	196	196
query9	8525	2369	2357	2357
query10	903	914	857	857
query11	16084	14954	14985	14954
query12	156	95	94	94
query13	1628	382	376	376
query14	10420	7911	7899	7899
query15	393	334	325	325
query16	7897	469	501	469
query17	1714	574	531	531
query18	2106	389	373	373
query19	230	206	202	202
query20	111	106	105	105
query21	201	104	97	97
query22	4270	4090	4072	4072
query23	33740	33239	32969	32969
query24	11875	2869	2927	2869
query25	639	367	371	367
query26	1772	151	151	151
query27	2910	315	281	281
query28	7263	1987	1973	1973
query29	1047	406	407	406
query30	313	149	148	148
query31	980	761	732	732
query32	95	55	56	55
query33	741	293	281	281
query34	936	464	488	464
query35	986	848	794	794
query36	1080	921	920	920
query37	200	84	81	81
query38	4294	4068	4065	4065
query39	1438	1364	1392	1364
query40	271	116	112	112
query41	45	43	44	43
query42	126	99	102	99
query43	513	461	484	461
query44	1293	736	751	736
query45	403	364	350	350
query46	1121	777	818	777
query47	1830	1762	1786	1762
query48	371	295	301	295
query49	1214	419	415	415
query50	823	403	415	403
query51	6775	6718	6685	6685
query52	112	87	93	87
query53	260	178	174	174
query54	960	445	435	435
query55	77	73	75	73
query56	270	240	242	240
query57	1149	1042	1052	1042
query58	232	226	232	226
query59	3063	2851	2925	2851
query60	293	260	267	260
query61	97	94	94	94
query62	844	643	652	643
query63	213	177	180	177
query64	10607	2420	1970	1970
query65	3261	3129	3151	3129
query66	1373	337	347	337
query67	15178	14800	14600	14600
query68	4586	533	556	533
query69	439	380	417	380
query70	1117	1151	1059	1059
query71	456	275	277	275
query72	19034	16485	16580	16485
query73	756	330	330	330
query74	9253	8713	8664	8664
query75	3593	2667	2661	2661
query76	3027	941	1021	941
query77	662	319	312	312
query78	10561	8922	8967	8922
query79	2230	518	513	513
query80	1659	500	554	500
query81	587	232	227	227
query82	737	138	137	137
query83	317	150	148	148
query84	265	84	76	76
query85	1772	294	290	290
query86	452	310	288	288
query87	4652	4485	4489	4485
query88	4326	2510	2513	2510
query89	399	286	288	286
query90	1809	198	197	197
query91	140	119	120	119
query92	63	51	52	51
query93	2348	530	523	523
query94	762	290	286	286
query95	348	313	262	262
query96	598	275	275	275
query97	3221	3089	3078	3078
query98	223	196	195	195
query99	1678	1240	1244	1240
Total cold run time: 317291 ms
Total hot run time: 202111 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3efca2d39730334ff210701fc8875a0ea35bf88a, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.69	0.08	0.07
query5	0.49	0.48	0.51
query6	1.12	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.50
query10	0.52	0.53	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.62	0.59	0.58
query14	0.77	0.79	0.78
query15	0.85	0.82	0.82
query16	0.36	0.37	0.35
query17	0.96	0.95	1.01
query18	0.23	0.21	0.21
query19	1.74	1.70	1.73
query20	0.02	0.01	0.01
query21	15.83	0.75	0.65
query22	3.84	7.79	1.88
query23	18.28	1.42	1.29
query24	2.04	0.24	0.22
query25	0.15	0.08	0.08
query26	0.29	0.20	0.20
query27	0.45	0.22	0.22
query28	13.29	1.02	0.99
query29	12.58	3.27	3.29
query30	0.25	0.06	0.06
query31	2.88	0.38	0.38
query32	3.27	0.48	0.48
query33	2.89	2.84	2.91
query34	17.16	4.37	4.35
query35	4.44	4.38	4.42
query36	0.65	0.49	0.48
query37	0.18	0.16	0.16
query38	0.15	0.15	0.14
query39	0.05	0.04	0.04
query40	0.14	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.66 s
Total hot run time: 30.47 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 8, 2024

PR approved by anyone and no changes requested.

@Jibing-Li Jibing-Li merged commit 2ea5aab into apache:master Aug 8, 2024
30 of 34 checks passed
@Jibing-Li Jibing-Li deleted the rowcount branch August 8, 2024 09:26
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@Jibing-Li @dataroaring
[improvement](statistics)Return -1 when external table row count is u…
c02977a 
…nknown. (#38990)

Return -1 when external table row count is unknown. 
Don't cache any row count value when loading row count for external
table get exception.
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@Jibing-Li @wyxxxcat
[improvement](statistics)Return -1 when external table row count is u…
74d0a03 
…nknown. (apache#38990)

Return -1 when external table row count is unknown. 
Don't cache any row count value when loading row count for external
table get exception.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@Jibing-Li @dataroaring
[improvement](statistics)Return -1 when external table row count is u…
ed85d96 
…nknown. (#38990)

Return -1 when external table row count is unknown. 
Don't cache any row count value when loading row count for external
table get exception.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@englefly englefly englefly approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged doing reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Jibing-Li @doris-robot @englefly @morrySnow @dataroaring