Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](nereids)filter estimation for slot=unknown #39592

Merged
merged 3 commits into from
Aug 26, 2024
Merged

[fix](nereids)filter estimation for slot=unknown #39592

merged 3 commits into from
Aug 26, 2024

Conversation

englefly
Copy link
Contributor

Proposed changes

detect new pattern: slot=unknown
suppose slot.ndv = 5, slot.row=100
expect filter result row is 20, but in master, the result is 0

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buidlall

@englefly
Copy link
Contributor Author

run buildall

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38755 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1b7fd418c434c8915b129f758c317ff5d3f91d1, data reload: false

------ Round 1 ----------------------------------
q1	17848	5350	4421	4421
q2	2061	219	211	211
q3	11729	1053	1203	1053
q4	10552	714	763	714
q5	7791	2891	2844	2844
q6	269	158	158	158
q7	1028	653	666	653
q8	9578	2139	2157	2139
q9	7393	6624	6630	6624
q10	7097	2243	2237	2237
q11	521	284	281	281
q12	430	263	272	263
q13	18514	3057	3073	3057
q14	308	273	261	261
q15	593	532	522	522
q16	521	409	412	409
q17	1003	661	773	661
q18	7579	6880	6901	6880
q19	1514	1039	1017	1017
q20	702	371	367	367
q21	4314	2925	2981	2925
q22	1148	1069	1058	1058
Total cold run time: 112493 ms
Total hot run time: 38755 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4381	4356	4320	4320
q2	418	317	303	303
q3	2939	2756	2720	2720
q4	1944	1743	1762	1743
q5	5691	5734	5745	5734
q6	249	151	158	151
q7	2316	1914	1834	1834
q8	3333	3487	3467	3467
q9	8875	8895	8891	8891
q10	3611	3319	3423	3319
q11	644	552	536	536
q12	870	673	697	673
q13	17169	3217	3297	3217
q14	333	312	305	305
q15	598	533	535	533
q16	520	481	490	481
q17	1857	1560	1547	1547
q18	8276	7967	7688	7688
q19	1800	1649	1704	1649
q20	2176	1912	1969	1912
q21	5708	5523	5530	5523
q22	1186	1114	1104	1104
Total cold run time: 74894 ms
Total hot run time: 57650 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199343 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1b7fd418c434c8915b129f758c317ff5d3f91d1, data reload: false

query1	1290	908	884	884
query2	6578	2079	1942	1942
query3	10780	4062	4170	4062
query4	59840	27498	23424	23424
query5	5489	665	708	665
query6	455	211	218	211
query7	5808	357	344	344
query8	569	441	445	441
query9	9061	2654	2662	2654
query10	542	329	334	329
query11	18041	15544	15642	15544
query12	166	132	131	131
query13	1529	432	425	425
query14	11351	7500	7565	7500
query15	239	195	201	195
query16	7312	525	546	525
query17	1139	646	637	637
query18	1837	348	371	348
query19	318	174	226	174
query20	148	140	145	140
query21	242	145	145	145
query22	4716	4436	4487	4436
query23	34372	34027	34066	34027
query24	6032	2961	2997	2961
query25	558	428	415	415
query26	693	181	178	178
query27	1689	308	303	303
query28	3945	2113	2107	2107
query29	687	447	440	440
query30	224	185	182	182
query31	1039	844	837	837
query32	103	78	79	78
query33	504	358	345	345
query34	904	515	514	514
query35	886	789	768	768
query36	1105	955	985	955
query37	165	112	105	105
query38	3975	3914	3939	3914
query39	1543	1488	1470	1470
query40	238	156	155	155
query41	139	135	139	135
query42	134	118	120	118
query43	551	503	506	503
query44	1131	797	809	797
query45	225	194	196	194
query46	1139	770	771	770
query47	1939	1859	1879	1859
query48	421	359	348	348
query49	920	588	585	585
query50	890	473	478	473
query51	7269	7062	7093	7062
query52	120	106	106	106
query53	298	223	222	222
query54	645	509	518	509
query55	90	91	91	91
query56	340	321	318	318
query57	1222	1171	1123	1123
query58	305	312	302	302
query59	3063	2927	3006	2927
query60	361	335	333	333
query61	150	149	145	145
query62	796	665	699	665
query63	269	235	233	233
query64	3083	1900	1853	1853
query65	3257	3232	3226	3226
query66	1046	697	688	688
query67	15557	15295	15357	15295
query68	4223	624	602	602
query69	463	328	321	321
query70	1188	1175	1147	1147
query71	456	329	326	326
query72	6464	2404	2154	2154
query73	821	362	369	362
query74	9121	8945	8934	8934
query75	3437	2771	2871	2771
query76	1640	1022	1095	1022
query77	671	458	437	437
query78	9970	9168	9094	9094
query79	1085	563	550	550
query80	886	609	618	609
query81	586	265	273	265
query82	302	158	160	158
query83	236	216	214	214
query84	285	100	97	97
query85	868	369	367	367
query86	345	322	326	322
query87	4455	4437	4332	4332
query88	3333	2550	2577	2550
query89	439	328	321	321
query90	1879	234	232	232
query91	154	134	128	128
query92	92	77	81	77
query93	1098	552	557	552
query94	763	332	339	332
query95	390	306	302	302
query96	606	287	287	287
query97	3241	3182	3159	3159
query98	274	230	224	224
query99	1921	1349	1305	1305
Total cold run time: 313164 ms
Total hot run time: 199343 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b1b7fd418c434c8915b129f758c317ff5d3f91d1, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.52	0.51	0.50
query6	1.14	0.75	0.72
query7	0.02	0.02	0.02
query8	0.06	0.06	0.06
query9	0.54	0.50	0.48
query10	0.55	0.55	0.56
query11	0.17	0.13	0.13
query12	0.16	0.13	0.13
query13	0.63	0.62	0.60
query14	0.77	0.80	0.78
query15	0.84	0.84	0.84
query16	0.38	0.37	0.39
query17	1.05	1.02	1.06
query18	0.22	0.21	0.21
query19	1.98	1.90	1.84
query20	0.01	0.02	0.01
query21	15.42	0.69	0.67
query22	3.77	6.60	2.56
query23	18.28	1.34	1.31
query24	2.09	0.24	0.23
query25	0.16	0.08	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.19	1.03	1.02
query29	12.60	3.34	3.32
query30	0.43	0.24	0.24
query31	2.81	0.42	0.41
query32	3.24	0.49	0.49
query33	3.03	2.97	2.99
query34	17.13	4.51	4.45
query35	4.46	4.47	4.47
query36	0.68	0.47	0.51
query37	0.21	0.19	0.18
query38	0.18	0.18	0.17
query39	0.07	0.06	0.06
query40	0.19	0.16	0.15
query41	0.12	0.07	0.07
query42	0.08	0.07	0.07
query43	0.07	0.06	0.06
Total cold run time: 109.62 s
Total hot run time: 32.17 s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@englefly englefly merged commit 6f6054d into apache:master Aug 26, 2024
31 of 36 checks passed
@englefly englefly deleted the slotEqualToUnknown branch August 26, 2024 03:39
englefly added a commit to englefly/incubator-doris that referenced this pull request Aug 26, 2024
@englefly
[fix](nereids)filter estimation for slot=unknown (apache#39592)
8ff68a8 
## Proposed changes
detect new pattern: slot=unknown
suppose slot.ndv = 5, slot.row=100
expect filter result row is 20, but in master, the result is 0

Issue Number: close #xxx

<!--Describe your changes.-->
@englefly englefly mentioned this pull request Aug 26, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 27, 2024
@englefly
[fix](nereids)filter estimation for slot=unknown #39592 (#39933)
aacd61a 
## Proposed changes
pick #39592
Issue Number: close #xxx

<!--Describe your changes.-->
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@englefly @dataroaring
[fix](nereids)filter estimation for slot=unknown (#39592)
ac52063 
## Proposed changes
detect new pattern: slot=unknown
suppose slot.ndv = 5, slot.row=100
expect filter result row is 20, but in master, the result is 0

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@xzj7019 xzj7019 xzj7019 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@englefly @doris-robot @starocean999 @xzj7019 @yiguolei @morrySnow