Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](meta) Switch meta serialization to gson 1 #36125

Merged
merged 3 commits into from
Jun 14, 2024
Merged

[improvement](meta) Switch meta serialization to gson 1 #36125

merged 3 commits into from
Jun 14, 2024

Conversation

iszhangpch
Copy link
Contributor

@iszhangpch iszhangpch commented Jun 11, 2024 •
edited
Loading

Proposed changes

Issue Number:

Switch meta serialization to gson
Contains the following classes：

JournalEntity
DatabaseInfo
PartitionPersistInfo
ConsistencyCheckInfo
Timestamp
BrokerMgr.ModifyBrokerInfo
LoadErrorHub.Param
TableInfo

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor

run buildall

@iszhangpch iszhangpch force-pushed the master branch 2 times, most recently from 432d403 to 9036c57 Compare June 12, 2024 02:57
@iszhangpch
Copy link
Contributor Author

run buildall

2 similar comments
@iszhangpch
Copy link
Contributor Author

run buildall

@iszhangpch
Copy link
Contributor Author

run buildall

@iszhangpch
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40402 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06ef3fa36f0f31daa746326663ef15718d80b5e4, data reload: false

------ Round 1 ----------------------------------
q1	5846	4331	4261	4261
q2	1458	186	194	186
q3	6890	1071	1121	1071
q4	1019	849	805	805
q5	2703	2702	2605	2605
q6	235	153	172	153
q7	1031	629	591	591
q8	2017	2032	2084	2032
q9	6565	6449	6464	6449
q10	3857	3743	3701	3701
q11	338	236	233	233
q12	385	231	220	220
q13	17856	2991	2976	2976
q14	279	210	217	210
q15	511	467	464	464
q16	483	376	384	376
q17	947	717	699	699
q18	7972	7500	7431	7431
q19	1616	1463	1464	1463
q20	760	301	315	301
q21	7705	3846	3866	3846
q22	374	335	329	329
Total cold run time: 70847 ms
Total hot run time: 40402 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4269	4274	4232	4232
q2	367	269	262	262
q3	2983	2691	2693	2691
q4	1860	1523	1582	1523
q5	5245	5250	5265	5250
q6	213	127	126	126
q7	2116	1702	1669	1669
q8	3186	3337	3303	3303
q9	8534	8594	8610	8594
q10	4119	3925	3853	3853
q11	593	497	496	496
q12	812	640	650	640
q13	3702	3250	3295	3250
q14	297	293	292	292
q15	536	478	477	477
q16	488	441	450	441
q17	1815	1522	1490	1490
q18	7840	7916	7624	7624
q19	2376	1522	1631	1522
q20	2044	1927	1835	1835
q21	5134	5024	4931	4931
q22	610	558	548	548
Total cold run time: 59139 ms
Total hot run time: 55049 ms

@dataroaring
Copy link
Contributor

run buildall

dataroaring
dataroaring previously approved these changes Jun 12, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39793 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e413cea41baee32a9cf6b2bebb801c45476ff70, data reload: false

------ Round 1 ----------------------------------
q1	17844	4688	4345	4345
q2	2032	189	192	189
q3	10543	1085	1177	1085
q4	10243	763	779	763
q5	7463	2698	2740	2698
q6	223	139	139	139
q7	969	609	602	602
q8	9223	2052	2044	2044
q9	8900	6453	6442	6442
q10	8946	3685	3733	3685
q11	453	238	234	234
q12	440	227	225	225
q13	17786	2968	2949	2949
q14	282	208	217	208
q15	529	488	467	467
q16	532	376	370	370
q17	977	675	638	638
q18	8050	7468	7351	7351
q19	4794	1497	1471	1471
q20	666	319	326	319
q21	4915	3238	3886	3238
q22	392	333	331	331
Total cold run time: 116202 ms
Total hot run time: 39793 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4381	4262	4251	4251
q2	383	298	276	276
q3	2950	2724	2708	2708
q4	1883	1619	1588	1588
q5	5229	5223	5250	5223
q6	225	126	129	126
q7	2096	1714	1727	1714
q8	3179	3305	3294	3294
q9	8256	8273	8265	8265
q10	3867	3675	3592	3592
q11	572	481	477	477
q12	770	575	590	575
q13	16491	2955	3003	2955
q14	291	275	259	259
q15	532	491	466	466
q16	475	422	426	422
q17	1776	1491	1489	1489
q18	7553	7467	7402	7402
q19	2372	1580	1534	1534
q20	1999	1748	1754	1748
q21	4898	4678	4719	4678
q22	628	531	531	531
Total cold run time: 70806 ms
Total hot run time: 53573 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172961 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e413cea41baee32a9cf6b2bebb801c45476ff70, data reload: false

query1	932	379	373	373
query2	6453	2529	2428	2428
query3	6645	209	211	209
query4	20401	17183	17355	17183
query5	4134	485	456	456
query6	248	164	161	161
query7	4596	311	297	297
query8	326	299	288	288
query9	8462	2362	2344	2344
query10	608	314	286	286
query11	10514	10037	10033	10033
query12	134	86	86	86
query13	1630	363	362	362
query14	9730	7629	7664	7629
query15	240	192	187	187
query16	8011	270	283	270
query17	1878	525	514	514
query18	2040	269	275	269
query19	202	160	155	155
query20	92	83	82	82
query21	209	133	127	127
query22	4326	3893	3993	3893
query23	33508	33019	33083	33019
query24	11849	2863	2822	2822
query25	653	359	356	356
query26	1776	150	150	150
query27	2904	324	331	324
query28	7311	2012	2014	2012
query29	1048	607	605	605
query30	295	151	147	147
query31	954	722	753	722
query32	98	52	58	52
query33	771	294	288	288
query34	941	464	469	464
query35	754	614	628	614
query36	1068	972	956	956
query37	179	71	72	71
query38	2899	2768	2726	2726
query39	845	803	793	793
query40	289	136	132	132
query41	59	55	53	53
query42	122	102	106	102
query43	585	558	583	558
query44	1250	747	742	742
query45	201	169	166	166
query46	1098	730	737	730
query47	1847	1782	1740	1740
query48	399	333	302	302
query49	1153	425	412	412
query50	765	396	381	381
query51	6740	6732	6638	6638
query52	102	92	95	92
query53	369	291	289	289
query54	1005	457	469	457
query55	77	76	72	72
query56	280	251	256	251
query57	1199	1028	1039	1028
query58	264	253	242	242
query59	3548	3427	3154	3154
query60	301	266	299	266
query61	91	87	90	87
query62	654	476	438	438
query63	322	295	301	295
query64	9860	2259	1747	1747
query65	3190	3143	3077	3077
query66	1395	331	331	331
query67	15435	15000	14867	14867
query68	4590	573	547	547
query69	520	423	461	423
query70	1052	1117	1133	1117
query71	405	276	274	274
query72	7175	6025	5116	5116
query73	755	325	325	325
query74	5971	5489	5504	5489
query75	3356	2671	2639	2639
query76	2621	911	923	911
query77	664	301	301	301
query78	10516	9742	9675	9675
query79	2105	525	516	516
query80	879	501	463	463
query81	558	215	216	215
query82	711	108	103	103
query83	265	173	169	169
query84	245	87	84	84
query85	1854	343	342	342
query86	476	304	325	304
query87	3267	3115	3138	3115
query88	4068	2443	2431	2431
query89	464	379	389	379
query90	1826	193	195	193
query91	130	100	100	100
query92	64	47	52	47
query93	2237	528	507	507
query94	1240	186	185	185
query95	398	307	316	307
query96	621	269	274	269
query97	3248	3056	2995	2995
query98	219	205	194	194
query99	1079	843	824	824
Total cold run time: 275843 ms
Total hot run time: 172961 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e413cea41baee32a9cf6b2bebb801c45476ff70, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.07	0.10
query5	0.50	0.49	0.50
query6	1.12	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.48
query10	0.54	0.53	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.60
query14	0.80	0.82	0.77
query15	0.83	0.83	0.80
query16	0.37	0.36	0.36
query17	0.96	1.03	0.99
query18	0.22	0.23	0.28
query19	1.93	1.75	1.68
query20	0.02	0.01	0.01
query21	15.42	0.65	0.65
query22	4.20	6.94	2.09
query23	18.30	1.33	1.36
query24	2.16	0.21	0.23
query25	0.17	0.08	0.10
query26	0.26	0.17	0.17
query27	0.08	0.08	0.09
query28	13.20	1.01	0.99
query29	12.64	3.26	3.18
query30	0.26	0.06	0.06
query31	2.86	0.38	0.38
query32	3.32	0.49	0.46
query33	2.92	2.93	2.93
query34	17.01	4.40	4.37
query35	4.44	4.50	4.50
query36	0.66	0.46	0.46
query37	0.19	0.15	0.15
query38	0.14	0.14	0.14
query39	0.04	0.04	0.03
query40	0.16	0.15	0.14
query41	0.10	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.43 s
Total hot run time: 30.66 s

@iszhangpch
Copy link
Contributor Author

run buildall

1 similar comment
@iszhangpch
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40389 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e413cea41baee32a9cf6b2bebb801c45476ff70, data reload: false

------ Round 1 ----------------------------------
q1	18046	4494	4343	4343
q2	2703	188	199	188
q3	11320	1113	1063	1063
q4	10858	791	801	791
q5	8179	2717	2606	2606
q6	225	142	146	142
q7	979	613	598	598
q8	9215	2055	2050	2050
q9	9011	6449	6482	6449
q10	8958	3709	3749	3709
q11	440	234	237	234
q12	435	234	233	233
q13	17763	2978	2983	2978
q14	268	222	218	218
q15	503	477	464	464
q16	507	387	392	387
q17	959	679	711	679
q18	8244	7461	7426	7426
q19	4834	1497	1389	1389
q20	650	295	318	295
q21	4865	3823	3938	3823
q22	388	333	324	324
Total cold run time: 119350 ms
Total hot run time: 40389 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4347	4214	4203	4203
q2	390	281	283	281
q3	2951	2765	2743	2743
q4	1862	1628	1619	1619
q5	5216	5299	5272	5272
q6	217	131	131	131
q7	2139	1785	1777	1777
q8	3179	3324	3326	3324
q9	8337	8326	8339	8326
q10	3861	3731	3679	3679
q11	585	494	486	486
q12	764	601	593	593
q13	16510	2990	3051	2990
q14	288	268	254	254
q15	515	471	468	468
q16	466	416	438	416
q17	1789	1498	1450	1450
q18	7555	7412	7409	7409
q19	3157	1709	1474	1474
q20	2011	1784	1759	1759
q21	4849	4637	4746	4637
q22	592	549	538	538
Total cold run time: 71580 ms
Total hot run time: 53829 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172023 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e413cea41baee32a9cf6b2bebb801c45476ff70, data reload: false

query1	927	376	362	362
query2	6450	2363	2403	2363
query3	6652	216	210	210
query4	19698	17185	17134	17134
query5	4169	463	453	453
query6	244	160	163	160
query7	4599	297	293	293
query8	312	293	290	290
query9	8390	2396	2347	2347
query10	599	302	276	276
query11	10517	9948	9985	9948
query12	135	88	83	83
query13	1642	367	376	367
query14	10059	7053	7553	7053
query15	226	193	187	187
query16	7763	277	273	273
query17	1621	555	531	531
query18	1780	277	280	277
query19	204	160	156	156
query20	91	88	83	83
query21	210	126	130	126
query22	4185	3980	3989	3980
query23	33542	33233	32954	32954
query24	12262	2769	2789	2769
query25	667	350	362	350
query26	1750	152	152	152
query27	2958	317	324	317
query28	7606	2033	2035	2033
query29	1110	630	608	608
query30	294	147	147	147
query31	948	729	743	729
query32	99	52	58	52
query33	761	291	277	277
query34	982	483	462	462
query35	742	606	627	606
query36	1109	910	965	910
query37	300	68	69	68
query38	2854	2717	2711	2711
query39	853	836	795	795
query40	272	125	123	123
query41	53	52	49	49
query42	114	95	98	95
query43	599	547	527	527
query44	1193	735	743	735
query45	193	168	169	168
query46	1083	734	698	698
query47	1848	1741	1775	1741
query48	370	301	325	301
query49	1199	410	411	410
query50	768	397	388	388
query51	6696	6657	6697	6657
query52	102	90	98	90
query53	354	295	290	290
query54	933	450	442	442
query55	76	73	77	73
query56	274	262	259	259
query57	1160	1041	1042	1041
query58	267	228	239	228
query59	3432	3304	3113	3113
query60	288	276	272	272
query61	95	90	98	90
query62	643	447	433	433
query63	322	289	294	289
query64	9879	2252	1750	1750
query65	3141	3116	3126	3116
query66	1348	336	337	336
query67	15254	14665	14654	14654
query68	4506	543	547	543
query69	452	299	306	299
query70	1107	1110	1095	1095
query71	398	278	281	278
query72	6963	5210	5634	5210
query73	755	331	329	329
query74	5988	5560	5506	5506
query75	3460	2660	2651	2651
query76	2803	927	924	924
query77	458	316	309	309
query78	10293	9994	9873	9873
query79	2563	517	513	513
query80	2111	455	454	454
query81	573	224	228	224
query82	864	103	107	103
query83	306	168	161	161
query84	267	89	87	87
query85	2241	336	366	336
query86	505	293	308	293
query87	3192	3063	3108	3063
query88	3879	2452	2450	2450
query89	476	375	387	375
query90	1817	187	193	187
query91	128	102	109	102
query92	60	49	51	49
query93	2546	522	495	495
query94	1255	189	185	185
query95	410	314	313	313
query96	596	268	269	268
query97	3225	3065	3050	3050
query98	217	198	195	195
query99	1229	845	843	843
Total cold run time: 276909 ms
Total hot run time: 172023 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e413cea41baee32a9cf6b2bebb801c45476ff70, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.23	0.04	0.05
query4	1.67	0.09	0.08
query5	0.50	0.48	0.50
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.48	0.50
query10	0.55	0.55	0.55
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.60
query14	0.77	0.78	0.79
query15	0.82	0.82	0.82
query16	0.37	0.35	0.37
query17	1.04	1.00	1.03
query18	0.22	0.26	0.26
query19	1.79	1.76	1.70
query20	0.02	0.01	0.01
query21	15.44	0.65	0.64
query22	4.25	7.91	1.38
query23	18.28	1.35	1.24
query24	2.09	0.22	0.21
query25	0.16	0.09	0.10
query26	0.24	0.17	0.17
query27	0.09	0.07	0.08
query28	13.34	1.02	0.99
query29	12.65	3.24	3.21
query30	0.26	0.06	0.06
query31	2.87	0.38	0.38
query32	3.28	0.47	0.47
query33	2.90	2.92	2.93
query34	17.27	4.38	4.40
query35	4.53	4.57	4.57
query36	0.65	0.45	0.47
query37	0.17	0.16	0.15
query38	0.14	0.14	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.73 s
Total hot run time: 30.04 s

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 12, 2024
@iszhangpch
Copy link
Contributor Author

run buildall

@iszhangpch
Copy link
Contributor Author

run buildall

dataroaring
dataroaring approved these changes Jun 13, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 13, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39770 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b8edb68e8701a54761858eaf247eda6d5aed9c87, data reload: false

------ Round 1 ----------------------------------
q1	17620	4352	4254	4254
q2	2025	187	192	187
q3	10498	1103	1029	1029
q4	10190	848	865	848
q5	7466	2669	2605	2605
q6	217	135	134	134
q7	974	603	591	591
q8	9223	2060	2034	2034
q9	8954	6476	6424	6424
q10	8915	3720	3707	3707
q11	450	232	235	232
q12	422	229	239	229
q13	17775	2997	2999	2997
q14	277	212	223	212
q15	522	483	490	483
q16	519	383	370	370
q17	956	683	749	683
q18	7989	7489	7492	7489
q19	4765	1375	1507	1375
q20	691	319	318	318
q21	4894	3244	3838	3244
q22	389	331	325	325
Total cold run time: 115731 ms
Total hot run time: 39770 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4393	4257	4295	4257
q2	380	285	276	276
q3	2982	2809	2896	2809
q4	1960	1720	1706	1706
q5	5594	5545	5494	5494
q6	216	126	128	126
q7	2162	1846	1885	1846
q8	3236	3394	3383	3383
q9	8643	8661	8768	8661
q10	4035	3967	3696	3696
q11	580	488	538	488
q12	819	629	631	629
q13	17140	3148	3178	3148
q14	301	308	276	276
q15	515	473	487	473
q16	490	431	442	431
q17	1819	1528	1472	1472
q18	8145	7797	7762	7762
q19	1830	1493	1520	1493
q20	3096	1924	1846	1846
q21	5052	4961	4838	4838
q22	611	530	549	530
Total cold run time: 73999 ms
Total hot run time: 55640 ms

@iszhangpch
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39808 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b4a7d79fe319cc4e273fa6a7ba1fc864c8ac337, data reload: false

------ Round 1 ----------------------------------
q1	17604	4430	4316	4316
q2	2029	197	196	196
q3	10416	1208	1007	1007
q4	10192	844	737	737
q5	7468	2704	2641	2641
q6	223	138	135	135
q7	959	619	590	590
q8	9211	2072	2087	2072
q9	8816	6518	6443	6443
q10	8981	3728	3735	3728
q11	447	245	238	238
q12	443	228	221	221
q13	17773	2949	2974	2949
q14	258	225	225	225
q15	532	475	494	475
q16	519	382	380	380
q17	970	736	720	720
q18	8071	7394	7411	7394
q19	2660	1555	1488	1488
q20	676	309	325	309
q21	4873	3201	3846	3201
q22	397	353	343	343
Total cold run time: 113518 ms
Total hot run time: 39808 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4347	4217	4241	4217
q2	375	278	286	278
q3	2953	2782	2738	2738
q4	1865	1619	1594	1594
q5	5228	5290	5298	5290
q6	211	127	126	126
q7	2096	1724	1705	1705
q8	3167	3325	3332	3325
q9	8345	8343	8397	8343
q10	3892	3684	3641	3641
q11	580	476	478	476
q12	796	608	575	575
q13	17516	2983	2981	2981
q14	300	275	253	253
q15	526	477	472	472
q16	472	420	420	420
q17	1781	1492	1479	1479
q18	7809	7555	7394	7394
q19	1732	1530	1506	1506
q20	1978	1787	1780	1780
q21	4854	4714	4604	4604
q22	607	540	544	540
Total cold run time: 71430 ms
Total hot run time: 53737 ms

@iszhangpch
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39988 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 376664f0208fbb9e1f11ecd918ba89829808381f, data reload: false

------ Round 1 ----------------------------------
q1	17622	4713	4349	4349
q2	2037	195	191	191
q3	10542	1158	974	974
q4	10234	963	877	877
q5	7528	2748	2662	2662
q6	222	137	136	136
q7	960	612	604	604
q8	9226	2098	2086	2086
q9	9050	6575	6505	6505
q10	9027	3829	3732	3732
q11	454	246	241	241
q12	431	232	225	225
q13	19032	2963	2974	2963
q14	278	229	219	219
q15	510	480	484	480
q16	524	378	382	378
q17	997	679	632	632
q18	8077	7413	7400	7400
q19	1688	1603	1463	1463
q20	648	313	319	313
q21	4876	3220	3935	3220
q22	401	338	355	338
Total cold run time: 114364 ms
Total hot run time: 39988 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4318	4246	4261	4246
q2	359	278	279	278
q3	2998	2781	2759	2759
q4	1904	1582	1652	1582
q5	5278	5301	5328	5301
q6	222	128	130	128
q7	2163	1692	1702	1692
q8	3281	3337	3335	3335
q9	8400	8401	8437	8401
q10	3913	3623	3673	3623
q11	595	493	494	493
q12	783	608	588	588
q13	16802	3002	2982	2982
q14	294	248	261	248
q15	524	482	458	458
q16	470	409	439	409
q17	1765	1527	1457	1457
q18	7646	7511	7354	7354
q19	6400	1636	1584	1584
q20	2017	1782	1749	1749
q21	4849	4811	4828	4811
q22	621	536	526	526
Total cold run time: 75602 ms
Total hot run time: 54004 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170101 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 376664f0208fbb9e1f11ecd918ba89829808381f, data reload: false

query1	939	395	377	377
query2	6480	2466	2419	2419
query3	6666	200	211	200
query4	19731	17262	17313	17262
query5	4144	473	470	470
query6	247	155	168	155
query7	4592	296	296	296
query8	312	279	309	279
query9	8754	2449	2478	2449
query10	613	302	279	279
query11	10670	10213	10070	10070
query12	146	94	86	86
query13	1663	378	370	370
query14	10042	6937	7513	6937
query15	261	188	196	188
query16	7955	283	273	273
query17	1878	555	554	554
query18	2025	285	325	285
query19	196	157	156	156
query20	92	88	79	79
query21	213	141	131	131
query22	4342	4090	4071	4071
query23	33713	33005	32952	32952
query24	10617	2825	2786	2786
query25	577	354	354	354
query26	707	152	151	151
query27	2235	311	328	311
query28	6038	2083	2083	2083
query29	869	607	594	594
query30	282	148	151	148
query31	941	737	750	737
query32	94	52	56	52
query33	667	277	277	277
query34	875	484	466	466
query35	734	616	620	616
query36	1106	949	923	923
query37	131	73	72	72
query38	2894	2762	2741	2741
query39	891	796	805	796
query40	198	123	125	123
query41	56	52	54	52
query42	116	99	100	99
query43	575	553	534	534
query44	1058	717	729	717
query45	201	165	168	165
query46	1084	728	737	728
query47	1900	1808	1775	1775
query48	375	300	299	299
query49	1043	405	412	405
query50	762	400	380	380
query51	6773	6711	6641	6641
query52	109	94	95	94
query53	356	293	284	284
query54	898	450	437	437
query55	72	71	77	71
query56	277	256	257	256
query57	1137	1023	1140	1023
query58	237	245	245	245
query59	3436	3288	3322	3288
query60	311	269	271	269
query61	92	89	88	88
query62	662	461	439	439
query63	320	294	281	281
query64	8596	2268	1717	1717
query65	3128	3128	3055	3055
query66	814	327	327	327
query67	15409	15086	15106	15086
query68	4544	542	550	542
query69	448	302	306	302
query70	1087	1061	1057	1057
query71	367	266	274	266
query72	7239	2756	2579	2579
query73	748	323	323	323
query74	5991	5626	5470	5470
query75	3371	2621	2685	2621
query76	2340	920	880	880
query77	467	295	296	295
query78	10310	9835	9915	9835
query79	2137	523	504	504
query80	1189	463	449	449
query81	581	218	220	218
query82	651	105	99	99
query83	225	174	167	167
query84	229	86	83	83
query85	1861	322	269	269
query86	486	286	294	286
query87	3258	3134	3108	3108
query88	4090	2443	2431	2431
query89	480	370	391	370
query90	1755	182	180	180
query91	123	100	97	97
query92	60	50	48	48
query93	2440	514	495	495
query94	1211	188	192	188
query95	400	309	321	309
query96	587	273	265	265
query97	3231	3024	3043	3024
query98	214	201	191	191
query99	1252	827	832	827
Total cold run time: 268699 ms
Total hot run time: 170101 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.58 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 376664f0208fbb9e1f11ecd918ba89829808381f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.47	0.49
query6	1.13	0.72	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.54	0.56	0.54
query11	0.14	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.77	0.79	0.78
query15	0.83	0.82	0.81
query16	0.38	0.37	0.38
query17	1.02	1.05	1.05
query18	0.23	0.26	0.24
query19	1.81	1.78	1.67
query20	0.01	0.01	0.02
query21	15.41	0.66	0.65
query22	4.20	7.69	1.80
query23	18.33	1.39	1.32
query24	2.15	0.22	0.23
query25	0.16	0.08	0.08
query26	0.26	0.18	0.17
query27	0.09	0.08	0.09
query28	13.18	1.01	1.00
query29	12.60	3.27	3.28
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.29	0.47	0.48
query33	2.83	2.92	2.93
query34	16.95	4.43	4.48
query35	4.45	4.46	4.53
query36	0.65	0.46	0.47
query37	0.19	0.16	0.17
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.18	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.15 s
Total hot run time: 30.58 s

dataroaring
dataroaring requested changes Jun 13, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FsBroker is wrong.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 13, 2024
@iszhangpch iszhangpch changed the title [improvement](meta) Switch meta serialization to gson [improvement](meta) Switch meta serialization to gson 1 Jun 14, 2024
@iszhangpch
Copy link
Contributor Author

run buildall

@iszhangpch
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39845 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed42df0d38ba7294c542567550803a22a4587890, data reload: false

------ Round 1 ----------------------------------
q1	18067	4498	4360	4360
q2	2955	209	200	200
q3	11167	1116	1188	1116
q4	10775	881	731	731
q5	7564	2718	2689	2689
q6	232	144	141	141
q7	967	632	608	608
q8	9620	2054	2081	2054
q9	8915	6494	6463	6463
q10	8972	3726	3660	3660
q11	441	242	242	242
q12	399	248	232	232
q13	17764	2986	2968	2968
q14	275	223	229	223
q15	520	472	465	465
q16	517	386	377	377
q17	964	698	668	668
q18	7917	7392	7374	7374
q19	5836	1563	1427	1427
q20	650	304	321	304
q21	5120	3211	3923	3211
q22	390	333	332	332
Total cold run time: 120027 ms
Total hot run time: 39845 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4359	4239	4267	4239
q2	372	263	269	263
q3	2982	2744	2710	2710
q4	1860	1623	1633	1623
q5	5263	5278	5284	5278
q6	216	130	128	128
q7	2126	1767	1714	1714
q8	3168	3325	3347	3325
q9	8363	8322	8372	8322
q10	3855	3609	3712	3609
q11	593	485	513	485
q12	763	582	605	582
q13	17517	3018	3000	3000
q14	292	268	266	266
q15	504	482	467	467
q16	465	426	421	421
q17	1754	1506	1485	1485
q18	7633	7582	7152	7152
q19	1708	1581	1606	1581
q20	1982	1816	1775	1775
q21	4928	4794	4701	4701
q22	628	538	545	538
Total cold run time: 71331 ms
Total hot run time: 53664 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170148 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed42df0d38ba7294c542567550803a22a4587890, data reload: false

query1	923	367	370	367
query2	6442	2370	2262	2262
query3	6660	207	207	207
query4	19782	17180	17210	17180
query5	4146	476	467	467
query6	254	160	152	152
query7	4588	292	293	292
query8	313	275	278	275
query9	8450	2415	2378	2378
query10	642	301	285	285
query11	10565	10102	10114	10102
query12	135	87	84	84
query13	1630	351	355	351
query14	9248	7492	7537	7492
query15	235	183	187	183
query16	7799	255	261	255
query17	1915	546	516	516
query18	1896	263	265	263
query19	192	158	150	150
query20	88	84	80	80
query21	205	125	123	123
query22	4268	4241	3972	3972
query23	33595	32973	32831	32831
query24	11842	2900	2910	2900
query25	693	366	376	366
query26	1779	159	154	154
query27	2934	325	323	323
query28	7358	2027	2034	2027
query29	1090	648	629	629
query30	290	153	152	152
query31	944	739	761	739
query32	93	60	62	60
query33	780	296	284	284
query34	955	471	458	458
query35	751	620	629	620
query36	1094	943	932	932
query37	188	74	71	71
query38	2916	2727	2736	2727
query39	865	851	791	791
query40	281	128	124	124
query41	56	50	52	50
query42	126	95	102	95
query43	583	536	540	536
query44	1160	735	736	735
query45	196	166	160	160
query46	1086	733	728	728
query47	1827	1782	1765	1765
query48	372	286	289	286
query49	1176	399	391	391
query50	760	374	390	374
query51	6812	6633	6657	6633
query52	108	94	93	93
query53	388	298	289	289
query54	998	441	431	431
query55	74	74	73	73
query56	288	259	253	253
query57	1121	1072	1031	1031
query58	246	255	251	251
query59	3303	3464	3508	3464
query60	296	266	264	264
query61	94	89	114	89
query62	644	441	444	441
query63	313	288	283	283
query64	9863	2219	1760	1760
query65	3160	3114	3113	3113
query66	1363	370	333	333
query67	15289	15054	15023	15023
query68	4607	529	523	523
query69	453	297	308	297
query70	1096	1119	1084	1084
query71	386	269	265	265
query72	7160	2800	2567	2567
query73	737	321	316	316
query74	6086	5403	5444	5403
query75	3373	2668	2641	2641
query76	2516	929	924	924
query77	456	286	299	286
query78	10409	9800	9826	9800
query79	2192	508	503	503
query80	1035	464	452	452
query81	556	220	214	214
query82	831	101	102	101
query83	266	164	164	164
query84	251	82	84	82
query85	1910	327	268	268
query86	497	281	313	281
query87	3299	3085	3045	3045
query88	4115	2351	2343	2343
query89	471	392	388	388
query90	1809	202	189	189
query91	123	98	97	97
query92	64	49	48	48
query93	2441	507	510	507
query94	1261	188	187	187
query95	399	302	316	302
query96	585	274	262	262
query97	3234	3030	2997	2997
query98	203	197	194	194
query99	1278	850	869	850
Total cold run time: 274625 ms
Total hot run time: 170148 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed42df0d38ba7294c542567550803a22a4587890, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.09	0.08
query5	0.49	0.48	0.50
query6	1.14	0.72	0.74
query7	0.02	0.02	0.02
query8	0.05	0.05	0.05
query9	0.55	0.48	0.48
query10	0.53	0.54	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.61	0.59
query14	0.76	0.80	0.78
query15	0.84	0.83	0.83
query16	0.36	0.37	0.36
query17	0.94	0.98	0.94
query18	0.21	0.24	0.24
query19	1.85	1.76	1.78
query20	0.01	0.01	0.01
query21	15.42	0.65	0.66
query22	4.53	6.28	2.54
query23	18.29	1.40	1.28
query24	2.17	0.22	0.21
query25	0.16	0.08	0.08
query26	0.27	0.18	0.17
query27	0.09	0.09	0.09
query28	13.25	1.04	1.00
query29	12.62	3.26	3.28
query30	0.26	0.07	0.06
query31	2.87	0.39	0.38
query32	3.26	0.47	0.49
query33	2.87	2.90	2.88
query34	17.20	4.39	4.48
query35	4.57	4.48	4.54
query36	0.64	0.48	0.46
query37	0.18	0.16	0.15
query38	0.17	0.14	0.14
query39	0.04	0.04	0.04
query40	0.18	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.9 s
Total hot run time: 31.23 s

dataroaring
dataroaring approved these changes Jun 14, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 14, 2024
@dataroaring dataroaring merged commit a89a358 into apache:master Jun 14, 2024
25 of 27 checks passed
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
@iszhangpch @dataroaring
[improvement](meta) Switch meta serialization to gson 1 (#36125)
869828c 
## Proposed changes

Issue Number: 

Switch meta serialization to gson
Contains the following classes：

~~JournalEntity~~
DatabaseInfo
~~PartitionPersistInfo~~
ConsistencyCheckInfo
Timestamp
BrokerMgr.ModifyBrokerInfo
LoadErrorHub.Param
TableInfo

---------

Co-authored-by: zhangpeicheng <zhangpeicheng@meituan.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman left review comments

@w41ter w41ter w41ter approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@iszhangpch @doris-robot @dataroaring @morningman @w41ter