Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Chore](pipeline) set PipelineFragmentContext::_timeout and adjust dump_pipeline_tasks infomation display #35328

Merged
merged 2 commits into from
May 24, 2024
Merged

[Chore](pipeline) set PipelineFragmentContext::_timeout and adjust dump_pipeline_tasks infomation display #35328

merged 2 commits into from
May 24, 2024

Conversation

BiteTheDDDDt
Copy link
Contributor

@BiteTheDDDDt BiteTheDDDDt commented May 24, 2024 •
edited
Loading

Proposed changes

set PipelineFragmentContext::_timeout and adjust dump_pipeline_tasks infomation display

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

Gabriel39
Gabriel39 previously approved these changes May 24, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.65% (9018/25295)
Line Coverage: 27.31% (74565/273061)
Region Coverage: 26.54% (38591/145430)
Branch Coverage: 23.40% (19686/84136)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3dc34ad30ede278e5f059dac3db86100b8c1f73b_3dc34ad30ede278e5f059dac3db86100b8c1f73b/report/index.html

@BiteTheDDDDt BiteTheDDDDt changed the title [Chore](pipeline) adjust dump_pipeline_tasks infomation display [Chore](pipeline) set PipelineFragmentContext::_timeout and adjust dump_pipeline_tasks infomation display May 24, 2024
@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41075 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 658868685694078f3ac37f67493c714a6b8138c6, data reload: false

------ Round 1 ----------------------------------
q1	17608	5135	4294	4294
q2	2017	182	192	182
q3	10554	1153	1167	1153
q4	10184	836	718	718
q5	7484	2798	2685	2685
q6	222	134	138	134
q7	977	623	613	613
q8	9226	2126	2077	2077
q9	9917	6655	6675	6655
q10	9197	3924	3865	3865
q11	452	240	239	239
q12	483	277	219	219
q13	17237	3273	3217	3217
q14	294	220	222	220
q15	507	476	465	465
q16	519	406	397	397
q17	1003	675	764	675
q18	8351	8021	7949	7949
q19	6271	1497	1499	1497
q20	623	317	321	317
q21	5249	4053	3232	3232
q22	345	277	272	272
Total cold run time: 118720 ms
Total hot run time: 41075 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4503	4897	4441	4441
q2	377	267	272	267
q3	3166	2914	2869	2869
q4	1975	1741	1755	1741
q5	5292	5468	5490	5468
q6	220	124	129	124
q7	2181	1813	1803	1803
q8	3228	3380	3384	3380
q9	8719	8594	8822	8594
q10	4079	3970	3802	3802
q11	588	476	483	476
q12	756	585	592	585
q13	15863	3112	3111	3111
q14	294	272	259	259
q15	511	472	493	472
q16	495	438	423	423
q17	1806	1543	1487	1487
q18	8212	7519	7430	7430
q19	4310	1545	1527	1527
q20	2020	1794	1765	1765
q21	12375	4916	4759	4759
q22	555	488	492	488
Total cold run time: 81525 ms
Total hot run time: 55271 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.65% (9018/25296)
Line Coverage: 27.31% (74582/273078)
Region Coverage: 26.54% (38599/145445)
Branch Coverage: 23.41% (19695/84146)
Coverage Report: http://coverage.selectdb-in.cc/coverage/658868685694078f3ac37f67493c714a6b8138c6_658868685694078f3ac37f67493c714a6b8138c6/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 167737 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 658868685694078f3ac37f67493c714a6b8138c6, data reload: false

query1	912	388	373	373
query2	6450	2297	2270	2270
query3	6657	209	208	208
query4	18881	17476	17336	17336
query5	4184	417	419	417
query6	245	153	165	153
query7	4580	295	296	295
query8	239	181	175	175
query9	8492	2361	2363	2361
query10	449	269	260	260
query11	10460	10087	10067	10067
query12	135	87	85	85
query13	1641	359	361	359
query14	8546	6160	7643	6160
query15	212	173	174	173
query16	7747	276	260	260
query17	1419	541	544	541
query18	1954	265	266	265
query19	199	152	152	152
query20	97	85	87	85
query21	196	128	137	128
query22	4217	3862	4010	3862
query23	33581	32815	33225	32815
query24	10027	2833	2827	2827
query25	583	343	354	343
query26	706	154	152	152
query27	2251	329	330	329
query28	5501	2025	2036	2025
query29	842	600	593	593
query30	247	172	170	170
query31	955	754	762	754
query32	92	54	52	52
query33	683	280	270	270
query34	1070	468	483	468
query35	705	602	607	602
query36	1085	940	945	940
query37	113	71	71	71
query38	2976	2825	2840	2825
query39	852	790	790	790
query40	197	126	124	124
query41	48	43	41	41
query42	104	95	101	95
query43	583	533	527	527
query44	1064	732	734	732
query45	181	157	161	157
query46	1065	716	673	673
query47	1864	1762	1765	1762
query48	368	298	291	291
query49	841	370	386	370
query50	771	411	389	389
query51	6908	6868	6767	6767
query52	104	94	86	86
query53	381	280	287	280
query54	869	421	420	420
query55	76	71	72	71
query56	271	257	241	241
query57	1114	1011	1039	1011
query58	220	199	208	199
query59	3344	3174	2945	2945
query60	264	250	249	249
query61	90	90	89	89
query62	599	436	471	436
query63	303	280	283	280
query64	8562	2166	1685	1685
query65	3150	3069	3087	3069
query66	781	325	317	317
query67	15437	14973	14781	14781
query68	4551	541	548	541
query69	437	264	267	264
query70	1162	1109	1092	1092
query71	430	275	287	275
query72	7875	2695	2573	2573
query73	723	330	320	320
query74	6025	5579	5692	5579
query75	3389	2638	2581	2581
query76	2800	974	941	941
query77	386	269	266	266
query78	10147	9792	9655	9655
query79	2468	508	511	508
query80	940	442	443	442
query81	556	250	240	240
query82	725	99	94	94
query83	241	171	169	169
query84	249	87	85	85
query85	1491	272	257	257
query86	490	304	302	302
query87	3267	3241	3082	3082
query88	3998	2352	2340	2340
query89	484	460	383	383
query90	2065	187	187	187
query91	120	95	95	95
query92	61	47	48	47
query93	1986	513	506	506
query94	1294	181	186	181
query95	398	304	303	303
query96	589	260	264	260
query97	3185	3026	2995	2995
query98	246	224	222	222
query99	1214	854	872	854
Total cold run time: 263732 ms
Total hot run time: 167737 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 658868685694078f3ac37f67493c714a6b8138c6, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.06
query5	0.51	0.50	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.78	0.78
query15	0.83	0.81	0.80
query16	0.36	0.37	0.35
query17	0.96	1.03	1.00
query18	0.22	0.23	0.24
query19	1.75	1.74	1.71
query20	0.01	0.01	0.01
query21	15.74	0.64	0.64
query22	3.71	7.44	2.16
query23	18.30	1.38	1.25
query24	1.54	0.38	0.22
query25	0.16	0.09	0.08
query26	0.26	0.17	0.16
query27	0.08	0.09	0.07
query28	13.21	1.01	0.99
query29	13.29	3.34	3.23
query30	0.24	0.05	0.06
query31	2.89	0.38	0.38
query32	3.30	0.46	0.47
query33	2.89	2.88	2.83
query34	17.18	4.46	4.42
query35	4.52	4.48	4.53
query36	0.74	0.49	0.46
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.13	0.14
query41	0.09	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 109.36 s
Total hot run time: 30.63 s

@BiteTheDDDDt BiteTheDDDDt merged commit 5747a8c into apache:master May 24, 2024
28 of 31 checks passed
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

dataroaring pushed a commit that referenced this pull request May 26, 2024
@BiteTheDDDDt @dataroaring
[Chore](pipeline) set PipelineFragmentContext::_timeout and adjust du…
dc5dcda 
…mp_pipeline_tasks infomation display (#35328)

set PipelineFragmentContext::_timeout and adjust dump_pipeline_tasks infomation display
seawinde pushed a commit to seawinde/doris that referenced this pull request May 27, 2024
@BiteTheDDDDt @seawinde
[Chore](pipeline) set PipelineFragmentContext::_timeout and adjust du…
881945b 
…mp_pipeline_tasks infomation display (apache#35328)

set PipelineFragmentContext::_timeout and adjust dump_pipeline_tasks infomation display
BiteTheDDDDt added a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Jun 7, 2024
@BiteTheDDDDt
[Chore](pipeline) set PipelineFragmentContext::_timeout and adjust du…
12b643a 
…mp_pipeline_tasks infomation display (apache#35328)

set PipelineFragmentContext::_timeout and adjust dump_pipeline_tasks infomation display
@BiteTheDDDDt BiteTheDDDDt mentioned this pull request Jun 7, 2024
Merged
morningman pushed a commit that referenced this pull request Jun 7, 2024
@BiteTheDDDDt
[Chore](pipeline) set PipelineFragmentContext::_timeout and adjust du…
4fc3d0c 
…mp_pipeline_tasks infomation display (#36023)

## Proposed changes
pick from #35328
@Gabriel39 Gabriel39 mentioned this pull request Aug 5, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 5, 2024
@Gabriel39
[Chore](pipeline) set PipelineFragmentContext::_timeout (#38890)
0711423 
## Proposed changes

Now we use `query_timeout` to set a timeout value for queries. But for
pipelineX engine, Doris do not use it so each query will not end before
EOS. This PR fix it.

pick #35328

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Gabriel39 Gabriel39 Gabriel39 approved these changes

@Mryange Mryange Mryange approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/2.1.6-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@BiteTheDDDDt @doris-robot @Gabriel39 @Mryange @morningman @yiguolei @dataroaring