Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement](balance) don't balance tablet which has unfinish alter job #39121

Merged
merged 1 commit into from
Aug 9, 2024

Conversation

yujun777
Copy link
Collaborator

@yujun777 yujun777 commented Aug 8, 2024

Improvement: don't balance tablets that have unfished alter job.

Also fix partition rebalancer may balance colocate tablets.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777
Copy link
Collaborator Author

yujun777 commented Aug 8, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 8, 2024
@yujun777
Copy link
Collaborator Author

yujun777 commented Aug 8, 2024

run buildall

@yujun777 yujun777 force-pushed the not-balance-alter-table branch from f8d5b53 to 26a616f Compare August 8, 2024 12:15
@yujun777
Copy link
Collaborator Author

yujun777 commented Aug 8, 2024

run buildall

@yujun777 yujun777 force-pushed the not-balance-alter-table branch from 26a616f to b08732b Compare August 8, 2024 12:28
@yujun777
Copy link
Collaborator Author

yujun777 commented Aug 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39612 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b08732ba206b49b9eb147380996452f14c5f08a0, data reload: false

------ Round 1 ----------------------------------
q1	17925	4509	4446	4446
q2	2459	226	183	183
q3	11413	1176	982	982
q4	10482	730	683	683
q5	7578	2535	2506	2506
q6	223	137	138	137
q7	969	622	633	622
q8	9379	1941	1912	1912
q9	8786	6540	6542	6540
q10	7005	2205	2194	2194
q11	452	246	257	246
q12	408	220	218	218
q13	17826	2953	3000	2953
q14	274	229	235	229
q15	519	463	490	463
q16	489	384	383	383
q17	969	723	709	709
q18	8088	7615	7353	7353
q19	6330	1060	976	976
q20	681	326	328	326
q21	5628	4631	4575	4575
q22	1085	1002	976	976
Total cold run time: 118968 ms
Total hot run time: 39612 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4439	4310	4300	4300
q2	382	275	271	271
q3	2871	2573	2580	2573
q4	1923	1583	1562	1562
q5	5238	5249	5218	5218
q6	227	133	133	133
q7	2064	1663	1660	1660
q8	3174	3374	3320	3320
q9	8407	8346	8301	8301
q10	3400	3127	3100	3100
q11	582	491	479	479
q12	758	623	583	583
q13	16668	2974	2969	2969
q14	292	285	259	259
q15	523	489	487	487
q16	474	415	403	403
q17	1795	1476	1479	1476
q18	7667	7707	7319	7319
q19	4246	1615	1648	1615
q20	2020	1779	1766	1766
q21	5204	5125	5082	5082
q22	1089	1031	969	969
Total cold run time: 73443 ms
Total hot run time: 53845 ms

Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Aug 8, 2024

PR approved by anyone and no changes requested.

@yujun777
Copy link
Collaborator Author

yujun777 commented Aug 9, 2024

run feut

@yujun777
Copy link
Collaborator Author

yujun777 commented Aug 9, 2024

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 40540 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b08732ba206b49b9eb147380996452f14c5f08a0, data reload: false

------ Round 1 ----------------------------------
q1	17830	4725	4543	4543
q2	2021	183	177	177
q3	10591	1216	1145	1145
q4	10141	725	714	714
q5	7530	2661	2650	2650
q6	241	140	139	139
q7	1040	624	605	605
q8	9225	1986	1995	1986
q9	9018	6752	6822	6752
q10	7137	2270	2307	2270
q11	497	242	237	237
q12	429	227	228	227
q13	17775	2972	3020	2972
q14	296	243	248	243
q15	564	497	499	497
q16	539	396	392	392
q17	1011	670	703	670
q18	8279	7479	7341	7341
q19	4787	1168	1063	1063
q20	667	343	348	343
q21	6020	4683	4584	4584
q22	1105	1072	990	990
Total cold run time: 116743 ms
Total hot run time: 40540 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4693	4558	4550	4550
q2	387	277	263	263
q3	2957	2750	2706	2706
q4	2070	1736	1708	1708
q5	5606	5722	5572	5572
q6	259	140	138	138
q7	2231	1758	1740	1740
q8	3458	3710	3657	3657
q9	8934	8870	8838	8838
q10	3587	3282	3344	3282
q11	611	506	522	506
q12	850	619	658	619
q13	17284	3189	3196	3189
q14	332	291	299	291
q15	573	515	521	515
q16	515	470	431	431
q17	1978	1630	1578	1578
q18	7692	7577	7429	7429
q19	3655	1832	1758	1758
q20	2229	1779	1765	1765
q21	5452	5350	5135	5135
q22	1107	986	1029	986
Total cold run time: 76460 ms
Total hot run time: 56656 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202342 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b08732ba206b49b9eb147380996452f14c5f08a0, data reload: false

query1	960	401	396	396
query2	6440	1885	1795	1795
query3	6617	209	215	209
query4	31498	23034	23115	23034
query5	4199	491	490	490
query6	283	179	191	179
query7	4582	300	294	294
query8	257	195	190	190
query9	8435	2372	2346	2346
query10	914	865	839	839
query11	17858	15054	14838	14838
query12	149	96	91	91
query13	1645	386	363	363
query14	10599	7895	7826	7826
query15	431	313	315	313
query16	7762	481	440	440
query17	1655	564	545	545
query18	2071	371	373	371
query19	250	226	197	197
query20	116	106	107	106
query21	212	98	106	98
query22	4215	4000	3900	3900
query23	33691	33085	33232	33085
query24	10949	2860	2899	2860
query25	631	391	382	382
query26	1168	158	153	153
query27	2416	279	287	279
query28	7263	1993	1986	1986
query29	894	422	424	422
query30	302	147	151	147
query31	997	754	757	754
query32	94	58	57	57
query33	764	294	284	284
query34	947	464	469	464
query35	961	835	825	825
query36	1096	899	919	899
query37	147	82	81	81
query38	4254	4117	4157	4117
query39	1459	1376	1367	1367
query40	213	127	117	117
query41	50	45	47	45
query42	111	98	100	98
query43	495	458	460	458
query44	1260	754	760	754
query45	395	362	357	357
query46	1108	821	778	778
query47	1844	1758	1776	1758
query48	383	297	308	297
query49	1061	425	442	425
query50	802	410	412	410
query51	6760	6667	6668	6667
query52	101	94	98	94
query53	260	183	183	183
query54	907	465	449	449
query55	78	78	77	77
query56	289	251	252	251
query57	1146	1079	1061	1061
query58	249	224	234	224
query59	2910	2760	2560	2560
query60	326	267	271	267
query61	119	113	114	113
query62	822	647	659	647
query63	216	185	182	182
query64	9691	2518	1994	1994
query65	3238	3159	3274	3159
query66	910	342	361	342
query67	15276	14789	14686	14686
query68	8473	554	573	554
query69	496	423	437	423
query70	1189	1152	1143	1143
query71	548	267	264	264
query72	20590	16876	17674	16876
query73	970	327	326	326
query74	9308	8783	8787	8783
query75	5068	2698	2686	2686
query76	5074	1021	926	926
query77	769	315	315	315
query78	9800	9177	8918	8918
query79	9377	521	528	521
query80	915	486	489	486
query81	592	238	231	231
query82	287	135	135	135
query83	332	140	150	140
query84	264	82	77	77
query85	980	333	290	290
query86	344	296	315	296
query87	4738	4527	4602	4527
query88	5109	2496	2502	2496
query89	503	292	293	292
query90	2106	189	191	189
query91	146	134	115	115
query92	65	51	51	51
query93	5962	533	537	533
query94	1073	297	285	285
query95	355	262	258	258
query96	609	274	273	273
query97	3219	3007	3052	3007
query98	215	198	192	192
query99	1517	1261	1259	1259
Total cold run time: 331194 ms
Total hot run time: 202342 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.77 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b08732ba206b49b9eb147380996452f14c5f08a0, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.05	0.04
query3	0.22	0.05	0.06
query4	1.67	0.08	0.08
query5	0.49	0.49	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.49
query10	0.55	0.54	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.61	0.58
query14	0.75	0.77	0.81
query15	0.87	0.83	0.82
query16	0.37	0.36	0.37
query17	1.02	0.97	1.03
query18	0.22	0.22	0.21
query19	1.91	1.74	1.82
query20	0.01	0.01	0.01
query21	15.41	0.76	0.66
query22	4.64	7.38	2.19
query23	18.27	1.41	1.17
query24	2.02	0.22	0.21
query25	0.14	0.08	0.07
query26	0.30	0.20	0.21
query27	0.46	0.23	0.22
query28	13.37	1.04	0.99
query29	12.63	3.31	3.28
query30	0.24	0.05	0.04
query31	2.88	0.40	0.40
query32	3.28	0.47	0.46
query33	2.94	2.90	2.93
query34	17.19	4.31	4.41
query35	4.37	4.40	4.42
query36	0.66	0.47	0.47
query37	0.19	0.16	0.15
query38	0.16	0.16	0.16
query39	0.05	0.03	0.04
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.05
Total cold run time: 110.43 s
Total hot run time: 30.77 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 0e23c2d into apache:master Aug 9, 2024
30 checks passed
Copy link
Contributor

github-actions bot commented Aug 9, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 9, 2024
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
…job (#39121)

Improvement:   don't balance tablets that have unfished alter job.

Also fix partition rebalancer may balance colocate tablets.
yujun777 added a commit to yujun777/doris that referenced this pull request Aug 12, 2024
…job (apache#39121)

Improvement:   don't balance tablets that have unfished alter job.

Also fix partition rebalancer may balance colocate tablets.
dataroaring pushed a commit that referenced this pull request Aug 13, 2024
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
…job (apache#39121)

Improvement:   don't balance tablets that have unfished alter job.

Also fix partition rebalancer may balance colocate tablets.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
…job (#39121)

Improvement:   don't balance tablets that have unfished alter job.

Also fix partition rebalancer may balance colocate tablets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged doing reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants