Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Export) Enhance removeOldExportJobs Logic #47604

Merged
merged 5 commits into from
Feb 18, 2025
Merged

[fix](Export) Enhance removeOldExportJobs Logic #47604

merged 5 commits into from
Feb 18, 2025
+81 −0

Conversation

BePPPower
Copy link
Contributor

What problem does this PR solve?

This pull request introduces improvements to the removeOldExportJobs() method in the ExportMgr class, ensuring that it efficiently manages the number of historical export jobs based on the configured limit. The following changes have been made:

Logic Enhancement:

  • The method now retains a maximum number of historical jobs, removing the oldest jobs only when the count exceeds the defined limit.

Unit Tests:

  • Added a comprehensive unit test for removeOldExportJobs() in the ExportMgrTest class.
  • The test verifies that the method correctly removes jobs based on their creation time and maintains the maximum allowed number of historical jobs.
  • Additional checks ensure that the jobs remaining after the removal process are the most recent ones.

These changes improve the efficiency and reliability of the job management system, ensuring that it adheres to the specified constraints while providing robust testing coverage.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31461 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit af0fe7459eadd13d71bfc273e3b09ea44e91d1bd, data reload: false

------ Round 1 ----------------------------------
q1	17584	5229	5071	5071
q2	2052	324	170	170
q3	10373	1325	704	704
q4	10228	1027	547	547
q5	7539	2452	2413	2413
q6	186	168	136	136
q7	894	750	599	599
q8	9298	1337	1074	1074
q9	4986	4601	4706	4601
q10	6811	2307	1893	1893
q11	472	276	260	260
q12	342	352	228	228
q13	17772	3701	3048	3048
q14	240	231	205	205
q15	503	472	454	454
q16	639	625	596	596
q17	559	878	344	344
q18	6864	6172	6250	6172
q19	1212	954	551	551
q20	316	325	187	187
q21	2839	2179	1901	1901
q22	357	324	307	307
Total cold run time: 102066 ms
Total hot run time: 31461 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5127	5113	5135	5113
q2	242	330	232	232
q3	2179	2654	2325	2325
q4	1479	1853	1388	1388
q5	4242	4116	4143	4116
q6	203	167	124	124
q7	1843	1858	1659	1659
q8	2657	2640	2571	2571
q9	7324	7275	7185	7185
q10	3059	3186	2750	2750
q11	573	511	486	486
q12	680	768	596	596
q13	3416	4050	3271	3271
q14	279	300	287	287
q15	513	474	467	467
q16	654	664	646	646
q17	1117	1605	1332	1332
q18	7563	7276	7301	7276
q19	807	806	873	806
q20	1965	1974	1912	1912
q21	5419	4925	4878	4878
q22	641	596	547	547
Total cold run time: 51982 ms
Total hot run time: 49967 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183861 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit af0fe7459eadd13d71bfc273e3b09ea44e91d1bd, data reload: false

query1	974	379	397	379
query2	6529	1872	1874	1872
query3	6807	219	221	219
query4	26143	23816	23775	23775
query5	4293	622	485	485
query6	290	198	178	178
query7	4618	524	308	308
query8	301	247	226	226
query9	8607	2497	2478	2478
query10	460	324	256	256
query11	15614	15004	15056	15004
query12	159	112	108	108
query13	1667	520	400	400
query14	9156	6055	6849	6055
query15	196	192	177	177
query16	7373	657	465	465
query17	1210	709	579	579
query18	1962	414	324	324
query19	198	185	158	158
query20	123	127	119	119
query21	212	126	105	105
query22	4268	4221	4270	4221
query23	33952	32976	33021	32976
query24	7726	2405	2379	2379
query25	544	450	388	388
query26	1227	269	155	155
query27	2127	483	329	329
query28	3996	2409	2356	2356
query29	785	560	418	418
query30	231	192	152	152
query31	931	838	803	803
query32	75	60	60	60
query33	569	379	305	305
query34	772	842	495	495
query35	800	827	728	728
query36	979	986	887	887
query37	124	103	79	79
query38	4106	4079	4130	4079
query39	1476	1430	1392	1392
query40	207	117	102	102
query41	53	56	55	55
query42	129	105	101	101
query43	510	514	481	481
query44	1261	787	774	774
query45	180	170	169	169
query46	854	1045	637	637
query47	1763	1818	1726	1726
query48	369	410	299	299
query49	781	495	426	426
query50	686	735	408	408
query51	4170	4160	4175	4160
query52	108	105	95	95
query53	227	254	183	183
query54	490	472	421	421
query55	80	79	79	79
query56	286	275	264	264
query57	1119	1120	1058	1058
query58	245	232	243	232
query59	2769	2804	2633	2633
query60	292	265	289	265
query61	123	115	140	115
query62	790	725	674	674
query63	221	192	185	185
query64	4368	1018	667	667
query65	3191	3126	3134	3126
query66	1137	430	318	318
query67	15914	15464	15404	15404
query68	7999	778	505	505
query69	471	287	308	287
query70	1202	1126	1114	1114
query71	394	299	272	272
query72	5749	3652	3724	3652
query73	729	743	348	348
query74	8881	9103	8801	8801
query75	3151	3187	2657	2657
query76	3150	1175	740	740
query77	458	362	276	276
query78	9812	10065	9276	9276
query79	2480	810	596	596
query80	692	526	453	453
query81	487	287	230	230
query82	667	156	117	117
query83	172	175	164	164
query84	232	91	79	79
query85	769	358	308	308
query86	328	300	270	270
query87	4486	4554	4380	4380
query88	3564	2151	2145	2145
query89	394	323	289	289
query90	1926	189	193	189
query91	140	144	109	109
query92	70	60	56	56
query93	1280	1017	598	598
query94	724	424	302	302
query95	354	279	258	258
query96	490	559	264	264
query97	2808	2887	2737	2737
query98	291	205	194	194
query99	1417	1408	1266	1266
Total cold run time: 269670 ms
Total hot run time: 183861 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit af0fe7459eadd13d71bfc273e3b09ea44e91d1bd, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.03
query3	0.22	0.07	0.07
query4	1.61	0.10	0.10
query5	0.40	0.41	0.38
query6	1.17	0.66	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.52
query10	0.56	0.57	0.58
query11	0.15	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.59	0.60
query14	2.73	2.73	2.71
query15	0.93	0.85	0.85
query16	0.38	0.36	0.38
query17	1.04	1.02	1.03
query18	0.21	0.20	0.19
query19	1.84	1.80	1.93
query20	0.01	0.01	0.02
query21	15.36	0.86	0.56
query22	0.74	1.32	0.67
query23	14.85	1.40	0.64
query24	7.14	1.60	0.63
query25	0.49	0.25	0.09
query26	0.61	0.16	0.13
query27	0.06	0.06	0.04
query28	9.33	0.88	0.43
query29	12.53	4.01	3.32
query30	0.27	0.09	0.06
query31	2.84	0.59	0.38
query32	3.22	0.54	0.47
query33	3.03	3.01	3.05
query34	15.65	5.10	4.59
query35	4.54	4.53	4.54
query36	0.68	0.49	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.6 s
Total hot run time: 30.47 s

@BePPPower
Copy link
Contributor Author

run buildall

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31371 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e4519fa55e2fb7d017a944268851b649b08b57e3, data reload: false

------ Round 1 ----------------------------------
q1	17581	5273	5045	5045
q2	2057	305	164	164
q3	10410	1307	705	705
q4	10212	1015	551	551
q5	7532	2386	2334	2334
q6	190	165	134	134
q7	905	749	617	617
q8	9298	1319	1066	1066
q9	4821	4687	4564	4564
q10	6808	2312	1853	1853
q11	488	270	243	243
q12	344	353	212	212
q13	17763	3646	3087	3087
q14	227	226	207	207
q15	499	464	447	447
q16	632	614	594	594
q17	575	857	346	346
q18	6829	6283	6212	6212
q19	1199	939	541	541
q20	315	318	200	200
q21	2776	2188	1948	1948
q22	357	320	301	301
Total cold run time: 101818 ms
Total hot run time: 31371 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5151	5058	5109	5058
q2	232	324	235	235
q3	2124	2654	2348	2348
q4	1409	1859	1365	1365
q5	4233	4116	4134	4116
q6	206	162	124	124
q7	1853	1827	1669	1669
q8	2607	2624	2582	2582
q9	7261	7110	7090	7090
q10	2986	3184	2810	2810
q11	590	507	484	484
q12	695	765	621	621
q13	3421	3988	3263	3263
q14	268	286	264	264
q15	507	466	452	452
q16	632	688	640	640
q17	1144	1574	1351	1351
q18	7570	7295	7244	7244
q19	791	824	799	799
q20	1971	2037	1832	1832
q21	5353	4898	4826	4826
q22	643	600	571	571
Total cold run time: 51647 ms
Total hot run time: 49744 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182935 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e4519fa55e2fb7d017a944268851b649b08b57e3, data reload: false

query1	986	378	353	353
query2	6512	1916	1938	1916
query3	6791	216	210	210
query4	26645	23237	23488	23237
query5	4334	626	466	466
query6	290	196	191	191
query7	4611	496	292	292
query8	296	232	226	226
query9	8611	2488	2475	2475
query10	482	309	258	258
query11	15423	15115	14881	14881
query12	154	105	106	105
query13	1666	493	379	379
query14	9091	6748	6308	6308
query15	211	206	176	176
query16	7126	657	460	460
query17	1140	687	544	544
query18	1939	382	299	299
query19	189	188	146	146
query20	115	115	115	115
query21	208	130	100	100
query22	4046	4177	4385	4177
query23	34458	33358	32864	32864
query24	7673	2371	2342	2342
query25	520	422	410	410
query26	1213	262	145	145
query27	2144	498	322	322
query28	3941	2383	2367	2367
query29	718	529	403	403
query30	233	179	161	161
query31	912	843	786	786
query32	72	63	59	59
query33	563	345	293	293
query34	784	843	479	479
query35	804	790	738	738
query36	964	987	881	881
query37	123	105	75	75
query38	4084	4120	4040	4040
query39	1452	1444	1392	1392
query40	204	110	96	96
query41	55	48	60	48
query42	119	106	105	105
query43	507	534	483	483
query44	1267	777	815	777
query45	181	169	160	160
query46	860	1049	625	625
query47	1760	1805	1714	1714
query48	365	408	297	297
query49	781	503	424	424
query50	687	729	415	415
query51	4175	4222	4053	4053
query52	111	104	89	89
query53	229	248	178	178
query54	497	486	408	408
query55	79	80	82	80
query56	249	279	248	248
query57	1142	1121	1072	1072
query58	239	236	239	236
query59	2808	2746	2638	2638
query60	271	268	251	251
query61	116	116	115	115
query62	827	713	666	666
query63	224	181	184	181
query64	4251	1013	764	764
query65	3221	3089	3100	3089
query66	1145	408	318	318
query67	15800	15622	15406	15406
query68	3228	792	515	515
query69	467	297	273	273
query70	1197	1031	1117	1031
query71	385	283	259	259
query72	5714	3674	3787	3674
query73	739	741	356	356
query74	9176	8865	8832	8832
query75	3150	3165	2664	2664
query76	3080	1163	754	754
query77	463	353	278	278
query78	9893	10093	9444	9444
query79	1381	833	580	580
query80	697	564	468	468
query81	500	281	238	238
query82	406	160	118	118
query83	171	165	151	151
query84	241	83	77	77
query85	747	346	303	303
query86	320	321	281	281
query87	4451	4420	4245	4245
query88	2843	2158	2173	2158
query89	400	317	275	275
query90	1776	196	191	191
query91	142	140	108	108
query92	60	61	62	61
query93	1165	995	565	565
query94	639	442	290	290
query95	347	262	256	256
query96	484	543	269	269
query97	2736	2837	2741	2741
query98	222	209	200	200
query99	1317	1381	1273	1273
Total cold run time: 262231 ms
Total hot run time: 182935 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e4519fa55e2fb7d017a944268851b649b08b57e3, data reload: false

query1	0.04	0.05	0.04
query2	0.08	0.03	0.03
query3	0.22	0.08	0.07
query4	1.63	0.11	0.10
query5	0.42	0.42	0.42
query6	1.16	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.53	0.51
query10	0.57	0.57	0.57
query11	0.15	0.10	0.11
query12	0.14	0.10	0.11
query13	0.61	0.60	0.60
query14	2.75	2.83	2.83
query15	0.93	0.84	0.84
query16	0.40	0.38	0.39
query17	1.05	1.02	1.00
query18	0.21	0.19	0.19
query19	1.90	1.82	2.01
query20	0.02	0.02	0.01
query21	15.35	0.92	0.55
query22	0.76	1.13	0.63
query23	14.99	1.36	0.66
query24	7.75	1.28	0.40
query25	0.55	0.19	0.17
query26	0.59	0.16	0.14
query27	0.05	0.05	0.05
query28	9.50	0.86	0.42
query29	12.53	4.04	3.32
query30	0.25	0.08	0.05
query31	2.83	0.58	0.38
query32	3.23	0.53	0.46
query33	2.97	2.98	3.01
query34	15.81	5.04	4.54
query35	4.46	4.46	4.56
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.67 s
Total hot run time: 30.3 s

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31535 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef9bdc53ad638ddf9142701f9ed4212ebf49f355, data reload: false

------ Round 1 ----------------------------------
q1	17579	5176	5104	5104
q2	2057	314	177	177
q3	10389	1329	741	741
q4	10223	1005	541	541
q5	7527	2461	2319	2319
q6	193	166	134	134
q7	905	757	621	621
q8	9316	1294	1118	1118
q9	4807	4543	4572	4543
q10	6837	2300	1902	1902
q11	494	290	265	265
q12	350	372	223	223
q13	17771	3674	3083	3083
q14	231	223	207	207
q15	512	476	463	463
q16	628	617	573	573
q17	566	889	351	351
q18	6725	6313	6092	6092
q19	1210	951	569	569
q20	310	337	201	201
q21	2873	2232	1986	1986
q22	366	335	322	322
Total cold run time: 101869 ms
Total hot run time: 31535 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5139	5182	5084	5084
q2	235	324	227	227
q3	2179	2661	2314	2314
q4	1447	1812	1380	1380
q5	4288	4185	4168	4168
q6	209	161	126	126
q7	1865	1876	1652	1652
q8	2638	2639	2534	2534
q9	7338	7080	7026	7026
q10	2998	3175	2748	2748
q11	585	506	493	493
q12	682	768	606	606
q13	3486	3994	3274	3274
q14	273	295	263	263
q15	505	472	455	455
q16	629	689	624	624
q17	1136	1623	1340	1340
q18	7500	7291	7225	7225
q19	863	859	880	859
q20	1968	2057	1880	1880
q21	5417	4930	4875	4875
q22	646	606	589	589
Total cold run time: 52026 ms
Total hot run time: 49742 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182679 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef9bdc53ad638ddf9142701f9ed4212ebf49f355, data reload: false

query1	964	388	379	379
query2	6520	1834	1803	1803
query3	6815	212	224	212
query4	26962	23171	23469	23171
query5	4337	683	509	509
query6	300	186	173	173
query7	4597	503	304	304
query8	301	251	242	242
query9	8612	2508	2513	2508
query10	469	318	266	266
query11	15980	15168	14832	14832
query12	163	105	103	103
query13	1654	495	385	385
query14	9911	6177	6167	6167
query15	204	194	168	168
query16	7250	630	450	450
query17	1179	718	567	567
query18	1969	413	319	319
query19	202	197	161	161
query20	124	117	121	117
query21	208	125	110	110
query22	4003	4168	4329	4168
query23	34513	33419	32983	32983
query24	8367	2437	2352	2352
query25	524	437	393	393
query26	1236	271	151	151
query27	2759	499	337	337
query28	4364	2370	2399	2370
query29	750	542	415	415
query30	232	189	149	149
query31	956	832	787	787
query32	70	62	61	61
query33	549	365	295	295
query34	785	848	482	482
query35	800	839	740	740
query36	979	1015	903	903
query37	122	102	75	75
query38	4118	4197	4106	4106
query39	1454	1382	1400	1382
query40	206	113	101	101
query41	53	53	79	53
query42	135	105	104	104
query43	492	518	468	468
query44	1290	779	763	763
query45	171	168	156	156
query46	858	1034	668	668
query47	1755	1779	1689	1689
query48	377	404	294	294
query49	770	502	412	412
query50	728	709	413	413
query51	4118	4204	4109	4109
query52	107	105	97	97
query53	225	261	183	183
query54	476	494	408	408
query55	78	79	82	79
query56	264	275	256	256
query57	1133	1127	1051	1051
query58	252	242	250	242
query59	2546	2518	2535	2518
query60	287	283	275	275
query61	118	113	113	113
query62	831	742	674	674
query63	223	194	188	188
query64	4325	1006	655	655
query65	3353	3127	3115	3115
query66	1118	418	322	322
query67	15858	15516	15295	15295
query68	4350	794	516	516
query69	478	316	275	275
query70	1225	1135	1105	1105
query71	397	300	269	269
query72	5868	3520	3807	3520
query73	750	731	340	340
query74	8858	9061	8871	8871
query75	3154	3357	2682	2682
query76	3266	1182	764	764
query77	460	384	282	282
query78	10143	10218	9262	9262
query79	1865	821	578	578
query80	1060	535	533	533
query81	530	281	237	237
query82	432	123	95	95
query83	174	165	151	151
query84	237	93	69	69
query85	759	345	342	342
query86	341	301	303	301
query87	4433	4522	4370	4370
query88	2804	2175	2135	2135
query89	397	308	282	282
query90	1729	188	188	188
query91	132	138	108	108
query92	62	59	58	58
query93	1185	1004	579	579
query94	688	414	299	299
query95	354	261	261	261
query96	480	550	266	266
query97	2772	2860	2713	2713
query98	228	200	203	200
query99	1292	1440	1232	1232
Total cold run time: 268352 ms
Total hot run time: 182679 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef9bdc53ad638ddf9142701f9ed4212ebf49f355, data reload: false

query1	0.03	0.04	0.04
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.61	0.11	0.10
query5	0.41	0.41	0.40
query6	1.17	0.65	0.66
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.61	0.52	0.53
query10	0.57	0.59	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.63	0.59	0.61
query14	2.71	2.73	2.71
query15	0.93	0.85	0.87
query16	0.38	0.38	0.39
query17	1.02	1.06	1.03
query18	0.21	0.19	0.20
query19	1.88	1.85	2.02
query20	0.02	0.02	0.02
query21	15.35	0.92	0.55
query22	0.77	1.20	0.77
query23	14.82	1.33	0.61
query24	11.51	1.54	0.41
query25	0.31	0.25	0.07
query26	0.96	0.19	0.16
query27	0.05	0.05	0.04
query28	6.19	0.80	0.42
query29	12.52	3.90	3.26
query30	0.25	0.09	0.05
query31	2.83	0.60	0.38
query32	3.23	0.54	0.46
query33	2.97	3.03	3.07
query34	15.76	5.18	4.56
query35	4.50	4.53	4.54
query36	0.67	0.49	0.48
query37	0.09	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.02 s
Total hot run time: 30.34 s

morningman
morningman reviewed Feb 14, 2025
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java Outdated
@@ -2547,6 +2547,10 @@ public long saveExportJob(CountingDataOutputStream dos, long checksum) throws IO
long curTime = System.currentTimeMillis();
List<ExportJob> jobs = exportMgr.getJobs().stream().filter(t -> !t.isExpired(curTime))
.collect(Collectors.toList());
jobs.sort(Comparator.comparingLong(ExportJob::getCreateTimeMs));
while (jobs.size() > Config.max_export_history_job_num) {
jobs.remove(0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. only job with finished or cancelled state can be removed.
  2. can do sort only if jobs.size() > Config.max_export_history_job_num

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31727 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d47c164d02c1562eb1c369cc115a0d7cd260e7b, data reload: false

------ Round 1 ----------------------------------
q1	17598	5220	5068	5068
q2	2044	295	168	168
q3	10437	1247	763	763
q4	10249	1009	590	590
q5	7838	2400	2285	2285
q6	184	166	131	131
q7	900	737	594	594
q8	9318	1287	1109	1109
q9	4796	4829	4815	4815
q10	6871	2294	1896	1896
q11	470	277	250	250
q12	350	361	221	221
q13	17785	3669	3055	3055
q14	226	223	210	210
q15	502	457	439	439
q16	618	608	570	570
q17	588	857	335	335
q18	6531	6207	6203	6203
q19	1653	956	540	540
q20	323	311	188	188
q21	2760	2403	1993	1993
q22	371	329	304	304
Total cold run time: 102412 ms
Total hot run time: 31727 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5165	5163	5122	5122
q2	236	318	230	230
q3	2131	2679	2314	2314
q4	1472	1856	1394	1394
q5	4236	4130	4163	4130
q6	211	178	127	127
q7	1835	1813	1704	1704
q8	2609	2557	2571	2557
q9	7135	7111	7163	7111
q10	2999	3234	2805	2805
q11	577	505	501	501
q12	690	767	636	636
q13	3504	3853	3298	3298
q14	278	300	267	267
q15	506	455	451	451
q16	642	685	645	645
q17	1155	1578	1332	1332
q18	7460	7305	7304	7304
q19	843	878	882	878
q20	1957	2019	1844	1844
q21	5538	4938	4724	4724
q22	594	591	571	571
Total cold run time: 51773 ms
Total hot run time: 49945 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190478 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d47c164d02c1562eb1c369cc115a0d7cd260e7b, data reload: false

query1	1320	936	956	936
query2	6349	1825	1801	1801
query3	10966	4521	4477	4477
query4	55100	24344	23078	23078
query5	5175	552	478	478
query6	368	202	202	202
query7	5124	513	294	294
query8	320	250	237	237
query9	6308	2497	2489	2489
query10	426	323	259	259
query11	15280	15160	15108	15108
query12	150	109	102	102
query13	1131	523	374	374
query14	10760	7040	6711	6711
query15	203	221	183	183
query16	6942	649	478	478
query17	1067	717	554	554
query18	1555	393	302	302
query19	200	199	163	163
query20	133	122	122	122
query21	208	132	114	114
query22	4340	4520	4270	4270
query23	34109	33290	33406	33290
query24	5623	2464	2484	2464
query25	453	475	410	410
query26	628	289	158	158
query27	1649	522	339	339
query28	2770	2435	2400	2400
query29	602	591	461	461
query30	236	204	160	160
query31	880	880	827	827
query32	88	68	61	61
query33	447	383	326	326
query34	773	877	511	511
query35	834	827	758	758
query36	957	1011	897	897
query37	125	112	80	80
query38	4235	4316	4455	4316
query39	1495	1462	1437	1437
query40	214	118	111	111
query41	51	54	52	52
query42	134	106	111	106
query43	512	535	478	478
query44	1328	797	787	787
query45	183	175	167	167
query46	919	1071	686	686
query47	1843	1883	1759	1759
query48	397	433	317	317
query49	720	596	448	448
query50	748	761	427	427
query51	4322	4330	4240	4240
query52	106	105	104	104
query53	239	264	194	194
query54	493	504	425	425
query55	84	80	92	80
query56	262	270	256	256
query57	1157	1187	1109	1109
query58	242	264	265	264
query59	2728	2857	2590	2590
query60	289	285	270	270
query61	123	118	139	118
query62	729	734	670	670
query63	241	198	203	198
query64	1349	1021	787	787
query65	3314	3161	3193	3161
query66	742	411	356	356
query67	15963	15819	15440	15440
query68	5898	762	512	512
query69	548	317	274	274
query70	1161	1144	1123	1123
query71	451	303	261	261
query72	5400	3731	3814	3731
query73	1109	767	344	344
query74	9030	9118	8794	8794
query75	3203	3173	2723	2723
query76	3936	1183	739	739
query77	592	388	291	291
query78	10106	10101	9336	9336
query79	1845	822	585	585
query80	674	568	456	456
query81	488	270	251	251
query82	293	132	95	95
query83	189	171	150	150
query84	291	95	73	73
query85	736	349	293	293
query86	352	313	287	287
query87	4444	4674	4462	4462
query88	2830	2186	2162	2162
query89	390	313	292	292
query90	1747	195	189	189
query91	127	136	105	105
query92	79	64	58	58
query93	2330	1012	576	576
query94	706	405	301	301
query95	348	264	260	260
query96	487	570	265	265
query97	2775	2812	2755	2755
query98	232	209	208	208
query99	1322	1376	1300	1300
Total cold run time: 292880 ms
Total hot run time: 190478 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.58 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d47c164d02c1562eb1c369cc115a0d7cd260e7b, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.42	0.41
query6	1.18	0.67	0.64
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.53
query10	0.58	0.57	0.58
query11	0.15	0.11	0.10
query12	0.15	0.10	0.11
query13	0.63	0.61	0.61
query14	2.73	2.72	2.70
query15	0.91	0.87	0.86
query16	0.38	0.38	0.38
query17	1.01	1.02	1.02
query18	0.21	0.19	0.20
query19	1.93	1.83	1.95
query20	0.01	0.02	0.01
query21	15.35	0.90	0.53
query22	0.74	1.32	0.94
query23	14.70	1.36	0.62
query24	11.94	1.12	0.42
query25	0.33	0.28	0.21
query26	1.03	0.19	0.14
query27	0.05	0.04	0.05
query28	5.91	0.78	0.44
query29	12.53	3.92	3.30
query30	0.25	0.09	0.06
query31	2.82	0.59	0.38
query32	3.23	0.54	0.47
query33	3.00	3.04	3.01
query34	15.80	5.10	4.50
query35	4.57	4.54	4.50
query36	0.67	0.50	0.50
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.33 s
Total hot run time: 30.58 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 18, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 6a898d7 into apache:master Feb 18, 2025
26 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x-experimental dev/3.0.x-experimental reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@BePPPower @Thearas @doris-robot @morningman @CalvinKirs