Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chore](agent) log the binary message size of agent tasks #43239

Merged
merged 2 commits into from
Nov 5, 2024
Merged

[chore](agent) log the binary message size of agent tasks #43239

merged 2 commits into from
Nov 5, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Nov 5, 2024

No description provided.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Nov 5, 2024

run buildall

@w41ter w41ter force-pushed the log_binary_message_size branch from 9ed1064 to d7c72ce Compare November 5, 2024 03:51
@w41ter
Copy link
Contributor Author

w41ter commented Nov 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41399 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d7c72ce00017795ca8ce26a008300b459982c41c, data reload: false

------ Round 1 ----------------------------------
q1	17581	7638	7303	7303
q2	2045	159	158	158
q3	10571	1135	1169	1135
q4	10206	904	869	869
q5	7744	3098	3067	3067
q6	241	153	151	151
q7	1009	596	594	594
q8	9352	2012	2046	2012
q9	6570	6467	6490	6467
q10	7031	2411	2454	2411
q11	463	269	265	265
q12	414	232	217	217
q13	17768	3009	3032	3009
q14	255	214	225	214
q15	569	536	509	509
q16	645	603	597	597
q17	982	592	523	523
q18	7432	6608	7072	6608
q19	1325	1043	1090	1043
q20	486	181	180	180
q21	4047	3055	3083	3055
q22	1126	1012	1022	1012
Total cold run time: 107862 ms
Total hot run time: 41399 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7300	7266	7301	7266
q2	328	221	221	221
q3	2877	2810	2743	2743
q4	1957	1714	1670	1670
q5	5481	5507	5530	5507
q6	220	140	141	140
q7	2136	1775	1689	1689
q8	3249	3433	3443	3433
q9	8645	8627	8607	8607
q10	3492	3453	3475	3453
q11	608	504	500	500
q12	834	563	580	563
q13	9858	3034	3024	3024
q14	302	256	261	256
q15	573	516	519	516
q16	674	650	624	624
q17	1848	1626	1596	1596
q18	7939	7413	7475	7413
q19	1687	1547	1464	1464
q20	2071	1820	1794	1794
q21	5480	5181	5282	5181
q22	1135	1025	1022	1022
Total cold run time: 68694 ms
Total hot run time: 58682 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191549 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d7c72ce00017795ca8ce26a008300b459982c41c, data reload: false

query1	962	372	401	372
query2	6535	2088	2067	2067
query3	6792	213	217	213
query4	34225	23527	23658	23527
query5	4278	455	443	443
query6	291	174	169	169
query7	4604	296	299	296
query8	286	233	226	226
query9	9529	2640	2634	2634
query10	497	251	275	251
query11	18065	15222	15220	15220
query12	146	105	97	97
query13	1664	428	400	400
query14	10379	6786	7314	6786
query15	287	181	182	181
query16	8047	461	465	461
query17	1755	562	537	537
query18	2119	298	281	281
query19	367	145	142	142
query20	116	103	105	103
query21	206	99	96	96
query22	4601	4309	4453	4309
query23	34819	34083	34149	34083
query24	11155	2727	2773	2727
query25	621	373	372	372
query26	1171	162	157	157
query27	2372	279	277	277
query28	7497	2438	2416	2416
query29	811	406	408	406
query30	316	155	162	155
query31	1042	805	824	805
query32	94	53	57	53
query33	781	264	262	262
query34	1006	509	537	509
query35	879	723	728	723
query36	1114	943	940	940
query37	143	70	71	70
query38	4380	4340	4204	4204
query39	1458	1431	1420	1420
query40	211	95	94	94
query41	46	44	44	44
query42	108	95	92	92
query43	537	499	500	499
query44	1272	814	808	808
query45	176	163	159	159
query46	1145	707	691	691
query47	1966	1816	1857	1816
query48	402	314	304	304
query49	1134	397	396	396
query50	806	395	389	389
query51	7427	7202	7062	7062
query52	102	82	85	82
query53	261	177	178	177
query54	1299	414	404	404
query55	84	76	74	74
query56	243	237	237	237
query57	1280	1176	1143	1143
query58	230	219	202	202
query59	3264	3007	2841	2841
query60	271	252	261	252
query61	122	121	119	119
query62	877	692	671	671
query63	215	187	182	182
query64	4293	733	711	711
query65	3320	3220	3252	3220
query66	1453	321	316	316
query67	16234	16143	15813	15813
query68	4524	568	574	568
query69	450	261	260	260
query70	1180	1163	1124	1124
query71	410	254	253	253
query72	6664	4172	4303	4172
query73	778	361	386	361
query74	10186	9105	8985	8985
query75	3492	2695	2723	2695
query76	2964	1086	1086	1086
query77	441	272	271	271
query78	10421	9581	9414	9414
query79	1244	591	611	591
query80	787	422	415	415
query81	549	237	240	237
query82	1308	121	120	120
query83	186	137	141	137
query84	238	71	71	71
query85	1082	312	284	284
query86	315	302	279	279
query87	4805	4693	4679	4679
query88	3178	2208	2201	2201
query89	401	292	291	291
query90	2079	187	190	187
query91	137	101	99	99
query92	59	49	55	49
query93	1094	537	548	537
query94	804	296	299	296
query95	349	249	249	249
query96	622	284	292	284
query97	2896	2680	2699	2680
query98	210	193	198	193
query99	1514	1306	1293	1293
Total cold run time: 299555 ms
Total hot run time: 191549 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d7c72ce00017795ca8ce26a008300b459982c41c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.22	0.07	0.07
query4	1.65	0.10	0.10
query5	0.42	0.40	0.40
query6	1.14	0.65	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.48	0.48
query10	0.55	0.55	0.54
query11	0.15	0.11	0.11
query12	0.16	0.11	0.11
query13	0.62	0.59	0.61
query14	2.68	2.87	2.78
query15	0.91	0.84	0.82
query16	0.39	0.38	0.38
query17	1.07	1.04	1.06
query18	0.23	0.22	0.22
query19	1.98	1.86	1.90
query20	0.01	0.01	0.01
query21	15.37	0.61	0.60
query22	2.30	2.44	2.41
query23	17.18	1.03	1.09
query24	2.84	1.51	1.21
query25	0.33	0.14	0.11
query26	0.43	0.14	0.13
query27	0.04	0.04	0.03
query28	10.16	1.10	1.06
query29	12.55	3.29	3.26
query30	0.24	0.06	0.06
query31	2.87	0.39	0.38
query32	3.27	0.47	0.46
query33	2.99	3.01	3.07
query34	16.99	4.48	4.49
query35	4.50	4.47	4.47
query36	0.66	0.49	0.50
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.13 s
Total hot run time: 33.75 s

@w41ter
Copy link
Contributor Author

w41ter commented Nov 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40966 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 559bbaa83275299d23073ec15d5c5be84e900f4e, data reload: false

------ Round 1 ----------------------------------
q1	17594	7357	7256	7256
q2	2070	170	148	148
q3	10637	1050	1187	1050
q4	10569	822	896	822
q5	7738	3024	3057	3024
q6	231	142	142	142
q7	993	600	605	600
q8	9345	1967	1994	1967
q9	6547	6372	6398	6372
q10	7103	2394	2445	2394
q11	463	262	260	260
q12	407	206	202	202
q13	17789	3022	2984	2984
q14	244	204	212	204
q15	561	526	515	515
q16	659	591	566	566
q17	966	489	520	489
q18	7132	6748	6702	6702
q19	1325	1038	980	980
q20	465	177	179	177
q21	3948	3108	3112	3108
q22	1089	1004	1005	1004
Total cold run time: 107875 ms
Total hot run time: 40966 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7306	7190	7214	7190
q2	330	227	228	227
q3	2884	2925	2948	2925
q4	2059	1831	1773	1773
q5	5731	5728	5722	5722
q6	222	140	140	140
q7	2249	1818	1819	1818
q8	3326	3545	3510	3510
q9	8968	8902	8878	8878
q10	3559	3513	3528	3513
q11	600	509	502	502
q12	835	661	645	645
q13	10555	3219	3190	3190
q14	302	272	286	272
q15	581	525	525	525
q16	685	649	632	632
q17	1830	1627	1643	1627
q18	8334	7774	7476	7476
q19	1708	1599	1533	1533
q20	2151	1913	1838	1838
q21	5624	5411	5427	5411
q22	1097	1052	1044	1044
Total cold run time: 70936 ms
Total hot run time: 60391 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195601 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 559bbaa83275299d23073ec15d5c5be84e900f4e, data reload: false

query1	1214	893	949	893
query2	6224	2080	2059	2059
query3	10816	4060	3959	3959
query4	67607	29194	23702	23702
query5	4997	444	444	444
query6	403	185	162	162
query7	5578	285	288	285
query8	307	232	235	232
query9	8900	2657	2629	2629
query10	444	251	247	247
query11	17562	15372	16006	15372
query12	162	102	106	102
query13	1619	455	411	411
query14	10469	7345	7838	7345
query15	209	190	183	183
query16	7056	453	459	453
query17	1027	551	552	551
query18	1778	296	296	296
query19	192	157	150	150
query20	120	113	111	111
query21	199	100	103	100
query22	4661	4142	4282	4142
query23	34485	34050	33900	33900
query24	5973	2720	2695	2695
query25	488	377	375	375
query26	647	151	151	151
query27	1685	280	283	280
query28	4101	2430	2394	2394
query29	663	417	404	404
query30	235	153	146	146
query31	988	762	798	762
query32	61	52	51	51
query33	384	269	260	260
query34	929	495	524	495
query35	817	738	722	722
query36	1074	944	978	944
query37	119	81	68	68
query38	4423	4297	4335	4297
query39	1452	1447	1415	1415
query40	199	95	94	94
query41	45	41	45	41
query42	104	97	92	92
query43	535	480	485	480
query44	1176	819	810	810
query45	179	163	161	161
query46	1115	691	675	675
query47	1933	1828	1825	1825
query48	430	324	320	320
query49	713	384	365	365
query50	808	379	394	379
query51	7226	7035	7029	7029
query52	97	87	86	86
query53	252	176	186	176
query54	521	403	388	388
query55	77	74	77	74
query56	253	237	228	228
query57	1304	1178	1149	1149
query58	205	195	214	195
query59	3166	3053	2875	2875
query60	265	239	241	239
query61	130	101	100	100
query62	796	669	648	648
query63	212	181	181	181
query64	1335	627	604	604
query65	3304	3222	3240	3222
query66	715	299	299	299
query67	15679	15520	15736	15520
query68	2570	562	600	562
query69	394	251	252	251
query70	1204	1123	1145	1123
query71	331	253	242	242
query72	4930	4079	4156	4079
query73	766	371	360	360
query74	9519	8974	9086	8974
query75	3403	2699	2683	2683
query76	1514	1118	1032	1032
query77	446	281	276	276
query78	10187	9375	9389	9375
query79	1372	577	596	577
query80	785	447	448	447
query81	513	241	244	241
query82	1284	121	118	118
query83	163	140	143	140
query84	280	78	73	73
query85	895	366	350	350
query86	334	310	312	310
query87	4817	4665	4753	4665
query88	3444	2220	2175	2175
query89	417	288	294	288
query90	2030	190	185	185
query91	185	99	98	98
query92	60	44	47	44
query93	1262	545	540	540
query94	756	284	290	284
query95	336	240	234	234
query96	611	272	285	272
query97	2886	2673	2677	2673
query98	211	198	197	197
query99	1563	1300	1291	1291
Total cold run time: 312310 ms
Total hot run time: 195601 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 559bbaa83275299d23073ec15d5c5be84e900f4e, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.22	0.06	0.06
query4	1.65	0.11	0.10
query5	0.42	0.40	0.41
query6	1.16	0.67	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.70	2.73	2.73
query15	0.89	0.82	0.81
query16	0.39	0.38	0.38
query17	1.06	1.04	1.03
query18	0.20	0.20	0.20
query19	1.90	1.84	1.85
query20	0.01	0.01	0.02
query21	15.37	0.61	0.60
query22	2.62	2.12	1.56
query23	17.23	0.97	0.73
query24	3.76	1.54	1.24
query25	0.12	0.09	0.12
query26	0.66	0.14	0.13
query27	0.05	0.06	0.04
query28	9.50	1.10	1.07
query29	12.54	3.24	3.19
query30	0.24	0.06	0.06
query31	2.89	0.39	0.38
query32	3.28	0.46	0.44
query33	2.98	3.01	3.04
query34	16.87	4.47	4.39
query35	4.46	4.48	4.50
query36	0.66	0.50	0.49
query37	0.09	0.05	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.45 s
Total hot run time: 32.33 s

dataroaring
dataroaring approved these changes Nov 5, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 5, 2024

PR approved by anyone and no changes requested.

@gavinchou gavinchou self-requested a review November 5, 2024 13:10
@gavinchou gavinchou merged commit da35d3d into apache:master Nov 5, 2024
27 of 30 checks passed
@w41ter w41ter deleted the log_binary_message_size branch November 6, 2024 06:19
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Nov 6, 2024
@w41ter
[chore](agent) log the binary message size of agent tasks (apache#43239)
52552f2
@w41ter w41ter mentioned this pull request Nov 6, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Nov 6, 2024
@w41ter
[chore](agent) log the binary message size of agent tasks (apache#43239)
3133205
@w41ter w41ter mentioned this pull request Nov 6, 2024
Merged
w41ter added a commit that referenced this pull request Nov 6, 2024
@w41ter
[chore](agent) log the binary message size of agent tasks #43239 (#43367
f0a9fed 
)

cherry pick from #43239
w41ter added a commit that referenced this pull request Nov 6, 2024
@w41ter
[chore](agent) log the binary message size of agent tasks #43239 (#43366
1f1bee3 
)

cherry pick from #43239
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Nov 11, 2024
@w41ter
[chore](agent) log the binary message size of agent tasks (apache#43239)
d292d83
@w41ter w41ter mentioned this pull request Nov 11, 2024
Merged
w41ter added a commit that referenced this pull request Nov 11, 2024
@w41ter
[chore](agent) log the binary message size of agent tasks #43239 (#43598
97ca900 
)

cherry pick from #43239
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.16-merged dev/2.1.8-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@w41ter @doris-robot @gavinchou @dataroaring