Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](Row store) get the correct value of row_store_page_size after FE restarts #38236

Merged
merged 39 commits into from
Jul 25, 2024

Conversation

lxr599
Copy link
Contributor

@lxr599 lxr599 commented Jul 23, 2024

Proposed changes

get the correct value of row_store_page_size after FE restarts

@doris-robot
Copy link

TPC-H: Total hot run time: 39743 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26e18ac623c12ffa189e17ac6897901ce8faa0c0, data reload: false

------ Round 1 ----------------------------------
q1	17650	4327	4280	4280
q2	2023	188	189	188
q3	10401	1169	1132	1132
q4	10187	815	708	708
q5	7533	2794	2647	2647
q6	218	139	139	139
q7	952	598	596	596
q8	9216	2081	2081	2081
q9	8777	6583	6559	6559
q10	8789	3778	3783	3778
q11	491	235	237	235
q12	397	227	222	222
q13	17873	2955	2988	2955
q14	273	234	234	234
q15	523	481	497	481
q16	521	368	375	368
q17	965	641	763	641
q18	8064	7592	7358	7358
q19	7370	1456	1311	1311
q20	661	329	326	326
q21	4915	3219	3309	3219
q22	351	286	285	285
Total cold run time: 118150 ms
Total hot run time: 39743 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4431	4231	4238	4231
q2	376	277	257	257
q3	2964	2884	2976	2884
q4	1972	1670	1762	1670
q5	5647	5530	5433	5433
q6	229	138	130	130
q7	2183	1842	1838	1838
q8	3246	3425	3453	3425
q9	8800	8825	8897	8825
q10	4138	3878	3708	3708
q11	592	516	502	502
q12	807	661	618	618
q13	16907	3167	3181	3167
q14	321	301	290	290
q15	518	500	478	478
q16	505	436	437	436
q17	1819	1520	1515	1515
q18	8131	8057	7828	7828
q19	1736	1666	1575	1575
q20	2936	1911	1855	1855
q21	8157	4894	4629	4629
q22	607	501	537	501
Total cold run time: 77022 ms
Total hot run time: 55795 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174056 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26e18ac623c12ffa189e17ac6897901ce8faa0c0, data reload: false

query1	908	368	378	368
query2	6432	1963	1848	1848
query3	6641	205	215	205
query4	26767	17694	17447	17447
query5	3811	491	503	491
query6	264	176	169	169
query7	4594	287	277	277
query8	227	190	184	184
query9	8534	2400	2390	2390
query10	432	318	280	280
query11	11798	10086	10112	10086
query12	111	81	80	80
query13	1635	362	357	357
query14	9445	7542	7698	7542
query15	217	175	173	173
query16	7738	499	473	473
query17	1364	538	535	535
query18	1907	276	272	272
query19	199	151	155	151
query20	92	86	82	82
query21	211	127	128	127
query22	4499	4310	3998	3998
query23	34172	33622	33669	33622
query24	10877	2938	2872	2872
query25	605	403	398	398
query26	705	149	148	148
query27	2252	276	289	276
query28	6193	2083	2055	2055
query29	867	634	633	633
query30	268	155	153	153
query31	986	779	770	770
query32	91	53	62	53
query33	640	356	347	347
query34	920	497	515	497
query35	865	733	763	733
query36	1141	978	948	948
query37	137	81	84	81
query38	2952	2830	2752	2752
query39	877	813	831	813
query40	200	121	117	117
query41	46	43	45	43
query42	112	99	97	97
query43	501	470	463	463
query44	1112	722	723	722
query45	190	159	160	159
query46	1101	728	756	728
query47	1844	1796	1798	1796
query48	353	291	296	291
query49	856	405	436	405
query50	772	381	389	381
query51	6776	6678	6763	6678
query52	98	95	91	91
query53	358	289	285	285
query54	861	456	445	445
query55	76	74	76	74
query56	285	265	268	265
query57	1146	1044	1044	1044
query58	237	252	273	252
query59	2938	2670	2559	2559
query60	322	273	265	265
query61	98	95	95	95
query62	798	640	651	640
query63	321	285	287	285
query64	9249	2324	1785	1785
query65	3167	3107	3109	3107
query66	740	321	319	319
query67	15583	15091	14853	14853
query68	4622	549	539	539
query69	730	426	349	349
query70	1154	1158	1061	1061
query71	411	278	276	276
query72	7692	5696	5484	5484
query73	775	342	333	333
query74	6432	6003	5721	5721
query75	3448	2693	2716	2693
query76	2684	953	933	933
query77	636	312	311	311
query78	9653	9202	10144	9202
query79	2993	517	536	517
query80	1360	464	467	464
query81	583	217	218	217
query82	736	137	132	132
query83	332	164	168	164
query84	271	86	84	84
query85	1666	321	300	300
query86	473	325	321	321
query87	3268	3106	3117	3106
query88	4060	2377	2385	2377
query89	472	395	382	382
query90	1729	193	196	193
query91	132	102	101	101
query92	58	48	49	48
query93	2387	512	519	512
query94	1132	278	304	278
query95	416	317	315	315
query96	600	274	276	274
query97	3170	3024	3002	3002
query98	230	199	205	199
query99	1637	1238	1270	1238
Total cold run time: 278456 ms
Total hot run time: 174056 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26e18ac623c12ffa189e17ac6897901ce8faa0c0, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.08	0.08
query5	0.50	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.48
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.14	0.13	0.12
query13	0.59	0.60	0.59
query14	0.75	0.77	0.78
query15	0.86	0.81	0.82
query16	0.36	0.36	0.37
query17	0.95	0.94	0.95
query18	0.23	0.23	0.22
query19	1.81	1.68	1.69
query20	0.01	0.01	0.01
query21	15.42	0.74	0.65
query22	4.66	6.96	1.99
query23	18.29	1.43	1.26
query24	2.09	0.22	0.22
query25	0.15	0.10	0.08
query26	0.29	0.21	0.21
query27	0.45	0.22	0.23
query28	13.32	1.02	1.00
query29	12.60	3.32	3.31
query30	0.25	0.05	0.06
query31	2.88	0.39	0.39
query32	3.26	0.49	0.46
query33	2.89	2.93	2.86
query34	16.96	4.37	4.33
query35	4.43	4.42	4.38
query36	0.65	0.50	0.46
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.14	0.12	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.97 s
Total hot run time: 30.53 s

@lxr599
Copy link
Contributor Author

lxr599 commented Jul 24, 2024

run p0

@lxr599
Copy link
Contributor Author

lxr599 commented Jul 24, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39831 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 297dc779c2e1a2e002e0041c33a09a620fb9e692, data reload: false

------ Round 1 ----------------------------------
q1	17661	4543	4268	4268
q2	2026	193	184	184
q3	10519	1167	1148	1148
q4	10248	794	847	794
q5	7529	2707	2648	2648
q6	221	142	141	141
q7	958	602	598	598
q8	9219	2056	2045	2045
q9	8729	6516	6549	6516
q10	8709	3825	3764	3764
q11	475	233	240	233
q12	389	231	232	231
q13	17777	2987	2988	2987
q14	283	230	235	230
q15	518	502	479	479
q16	499	390	385	385
q17	958	707	750	707
q18	8204	7350	7314	7314
q19	8193	1409	1366	1366
q20	661	324	332	324
q21	5009	3189	3269	3189
q22	355	280	286	280
Total cold run time: 119140 ms
Total hot run time: 39831 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4397	4264	4204	4204
q2	364	264	260	260
q3	3021	2890	2979	2890
q4	1943	1699	1713	1699
q5	5532	5550	5444	5444
q6	220	137	136	136
q7	2286	1899	1879	1879
q8	3295	3433	3399	3399
q9	8828	8898	8792	8792
q10	4165	3676	3863	3676
q11	618	523	508	508
q12	803	666	636	636
q13	17191	3162	3226	3162
q14	316	294	287	287
q15	531	482	496	482
q16	482	467	440	440
q17	1795	1488	1494	1488
q18	8104	7953	7757	7757
q19	1763	1564	1521	1521
q20	3094	1889	1874	1874
q21	5101	4765	4746	4746
q22	678	537	489	489
Total cold run time: 74527 ms
Total hot run time: 55769 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174307 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 297dc779c2e1a2e002e0041c33a09a620fb9e692, data reload: false

query1	917	382	377	377
query2	6423	1937	1821	1821
query3	6640	209	216	209
query4	28374	17681	17390	17390
query5	3642	487	495	487
query6	281	177	171	171
query7	4611	295	290	290
query8	258	196	194	194
query9	8373	2405	2404	2404
query10	438	306	276	276
query11	11325	9979	10111	9979
query12	117	91	90	90
query13	1651	380	370	370
query14	10422	7984	7540	7540
query15	214	157	168	157
query16	7887	493	473	473
query17	1586	544	520	520
query18	2015	271	277	271
query19	193	147	151	147
query20	91	83	81	81
query21	198	123	129	123
query22	4338	4033	4105	4033
query23	34233	33657	33632	33632
query24	11057	2898	2889	2889
query25	620	408	401	401
query26	1223	161	153	153
query27	2409	286	280	280
query28	7197	2071	2049	2049
query29	881	651	606	606
query30	251	152	162	152
query31	952	747	767	747
query32	98	53	54	53
query33	761	330	338	330
query34	927	507	504	504
query35	832	774	761	761
query36	1133	990	981	981
query37	160	90	81	81
query38	2982	2922	2862	2862
query39	871	863	836	836
query40	200	122	118	118
query41	45	44	46	44
query42	120	96	96	96
query43	495	464	464	464
query44	1204	727	720	720
query45	206	175	175	175
query46	1088	731	753	731
query47	1837	1799	1778	1778
query48	374	290	294	290
query49	842	399	410	399
query50	774	391	399	391
query51	6779	6712	6721	6712
query52	106	96	89	89
query53	351	294	282	282
query54	918	453	451	451
query55	77	77	74	74
query56	302	269	284	269
query57	1137	1058	1042	1042
query58	246	247	269	247
query59	2752	2542	2562	2542
query60	309	268	269	268
query61	97	95	93	93
query62	821	625	659	625
query63	325	288	283	283
query64	9430	2257	1723	1723
query65	3162	3099	3100	3099
query66	747	328	337	328
query67	15397	14866	14882	14866
query68	4457	537	557	537
query69	463	341	339	339
query70	1134	1137	1169	1137
query71	422	279	285	279
query72	7106	6438	5733	5733
query73	752	328	330	328
query74	6098	5740	5752	5740
query75	3430	2699	2715	2699
query76	2717	927	936	927
query77	419	298	304	298
query78	9821	9141	9166	9141
query79	2488	532	523	523
query80	2186	471	471	471
query81	607	224	223	223
query82	996	140	136	136
query83	304	164	228	164
query84	264	87	95	87
query85	1204	310	298	298
query86	462	286	333	286
query87	3262	3130	3092	3092
query88	3385	2363	2366	2363
query89	489	385	378	378
query90	1735	196	193	193
query91	137	104	102	102
query92	67	51	49	49
query93	1909	522	519	519
query94	1101	305	288	288
query95	411	321	315	315
query96	590	271	274	271
query97	3196	3023	3098	3023
query98	225	198	198	198
query99	1568	1251	1274	1251
Total cold run time: 280127 ms
Total hot run time: 174307 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.95 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 297dc779c2e1a2e002e0041c33a09a620fb9e692, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.09	0.11
query5	0.52	0.47	0.47
query6	1.15	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.55	0.48	0.48
query10	0.54	0.53	0.53
query11	0.16	0.11	0.12
query12	0.14	0.12	0.13
query13	0.60	0.59	0.59
query14	0.77	0.79	0.78
query15	0.86	0.82	0.82
query16	0.37	0.37	0.36
query17	0.95	0.95	0.97
query18	0.24	0.22	0.21
query19	1.81	1.71	1.74
query20	0.01	0.02	0.01
query21	15.47	0.78	0.64
query22	4.22	6.46	2.28
query23	18.27	1.38	1.28
query24	2.18	0.24	0.22
query25	0.15	0.08	0.08
query26	0.30	0.21	0.22
query27	0.46	0.24	0.23
query28	13.13	1.04	1.02
query29	12.63	3.34	3.32
query30	0.25	0.06	0.05
query31	2.88	0.39	0.40
query32	3.26	0.50	0.47
query33	2.93	2.86	2.91
query34	16.96	4.33	4.36
query35	4.39	4.37	4.43
query36	0.66	0.50	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.67 s
Total hot run time: 30.95 s

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 24, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit 7795533 into apache:master Jul 25, 2024
27 of 30 checks passed
eldenmoon pushed a commit that referenced this pull request Jul 25, 2024
@xiaokang xiaokang added the doing label Jul 28, 2024
xiaokang pushed a commit that referenced this pull request Aug 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged meta-change reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants