Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chore](sql) Forbid show hidden columns and create table with hidden column #38796

Merged
merged 8 commits into from
Aug 6, 2024
Merged

[chore](sql) Forbid show hidden columns and create table with hidden column #38796

merged 8 commits into from
Aug 6, 2024

Conversation

924060929
Copy link
Contributor

Proposed changes

Forbid show hidden columns and create table with hidden column

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@924060929
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the doing label Aug 2, 2024
@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

1 similar comment
@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41466 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10a629f56764a808e0c81d3b74344d897171d0e7, data reload: false

------ Round 1 ----------------------------------
q1	19495	4219	4114	4114
q2	2451	206	197	197
q3	11274	1259	1341	1259
q4	11878	850	931	850
q5	7652	2974	2946	2946
q6	219	137	137	137
q7	1061	628	611	611
q8	9423	1896	1927	1896
q9	8437	6571	6619	6571
q10	8697	3833	3799	3799
q11	422	251	242	242
q12	411	226	219	219
q13	17760	2959	2918	2918
q14	272	242	243	242
q15	522	477	494	477
q16	488	397	382	382
q17	954	914	898	898
q18	7977	7308	7271	7271
q19	1385	1208	1204	1204
q20	566	309	351	309
q21	5233	4641	4679	4641
q22	351	283	286	283
Total cold run time: 116928 ms
Total hot run time: 41466 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4012	3980	4005	3980
q2	330	232	219	219
q3	2969	2978	3004	2978
q4	1851	1851	1854	1851
q5	5218	5194	5192	5192
q6	212	128	127	127
q7	2042	1688	1677	1677
q8	3133	3212	3233	3212
q9	8253	8159	8232	8159
q10	3749	3833	3813	3813
q11	536	456	448	448
q12	707	553	528	528
q13	11188	2936	2930	2930
q14	286	249	255	249
q15	515	481	476	476
q16	455	391	396	391
q17	1712	1699	1677	1677
q18	7624	7366	7210	7210
q19	1667	1665	1662	1662
q20	1947	1745	1746	1745
q21	5368	5161	5062	5062
q22	529	461	460	460
Total cold run time: 64303 ms
Total hot run time: 54046 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 167861 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 10a629f56764a808e0c81d3b74344d897171d0e7, data reload: false

query1	911	378	365	365
query2	6488	1603	1634	1603
query3	6662	212	218	212
query4	20119	17401	17144	17144
query5	4271	512	534	512
query6	303	174	180	174
query7	4605	312	300	300
query8	257	200	192	192
query9	8529	2364	2367	2364
query10	431	272	262	262
query11	10440	10055	10031	10031
query12	134	86	86	86
query13	1631	369	372	369
query14	8669	6703	7446	6703
query15	205	157	161	157
query16	7046	455	417	417
query17	935	562	549	549
query18	1888	274	271	271
query19	183	146	142	142
query20	90	85	89	85
query21	205	98	95	95
query22	4179	3993	4011	3993
query23	33727	32853	32976	32853
query24	10333	2977	3050	2977
query25	676	378	382	378
query26	1812	149	149	149
query27	2988	272	273	272
query28	7022	1963	1947	1947
query29	1310	411	410	410
query30	286	153	150	150
query31	906	751	749	749
query32	99	55	58	55
query33	725	308	311	308
query34	910	474	484	474
query35	854	742	738	738
query36	994	849	853	849
query37	294	78	78	78
query38	2868	2740	2745	2740
query39	878	810	815	810
query40	289	115	112	112
query41	50	43	44	43
query42	127	102	108	102
query43	460	421	410	410
query44	1191	726	725	725
query45	206	179	178	178
query46	1083	813	771	771
query47	1811	1702	1694	1694
query48	363	293	294	293
query49	1188	435	428	428
query50	886	435	422	422
query51	6819	6631	6702	6631
query52	101	95	87	87
query53	259	183	185	183
query54	626	453	462	453
query55	75	75	75	75
query56	279	261	273	261
query57	1158	1035	1052	1035
query58	311	278	310	278
query59	2688	2483	2331	2331
query60	302	272	283	272
query61	96	93	115	93
query62	926	657	659	657
query63	211	180	179	179
query64	5877	1911	1906	1906
query65	3149	3111	3090	3090
query66	1442	339	327	327
query67	15337	14690	14734	14690
query68	4318	574	592	574
query69	439	307	309	307
query70	1071	1029	1057	1029
query71	413	280	271	271
query72	7104	2715	2504	2504
query73	746	333	329	329
query74	6048	5614	5605	5605
query75	3350	2705	2727	2705
query76	2513	1206	1275	1206
query77	436	317	316	316
query78	9340	8937	8902	8902
query79	1295	530	517	517
query80	1008	500	500	500
query81	549	225	226	225
query82	1079	139	131	131
query83	245	164	167	164
query84	279	83	82	82
query85	1342	364	362	362
query86	398	278	323	278
query87	3228	3111	3080	3080
query88	3082	2494	2527	2494
query89	389	304	289	289
query90	1792	191	188	188
query91	122	98	98	98
query92	62	50	48	48
query93	1370	612	616	612
query94	900	293	261	261
query95	366	259	261	259
query96	596	282	280	280
query97	3266	3062	3064	3062
query98	213	199	196	196
query99	1595	1300	1298	1298
Total cold run time: 261127 ms
Total hot run time: 167861 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10a629f56764a808e0c81d3b74344d897171d0e7, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.05
query3	0.24	0.05	0.04
query4	1.67	0.06	0.07
query5	0.48	0.48	0.48
query6	1.13	0.72	0.71
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.57	0.51	0.50
query10	0.57	0.56	0.56
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.61	0.59
query14	0.78	0.78	0.81
query15	0.89	0.87	0.85
query16	0.36	0.35	0.36
query17	1.00	0.98	1.03
query18	0.22	0.22	0.21
query19	1.81	1.73	1.71
query20	0.01	0.01	0.01
query21	15.44	0.76	0.67
query22	4.00	7.36	1.17
query23	17.65	1.40	1.33
query24	2.31	0.22	0.22
query25	0.17	0.07	0.08
query26	0.32	0.22	0.21
query27	0.46	0.24	0.23
query28	13.17	1.00	0.96
query29	12.68	3.34	3.36
query30	0.26	0.06	0.04
query31	2.89	0.40	0.41
query32	3.22	0.49	0.48
query33	2.94	2.97	2.96
query34	15.41	4.27	4.28
query35	4.30	4.30	4.35
query36	0.67	0.48	0.48
query37	0.19	0.16	0.16
query38	0.16	0.16	0.15
query39	0.04	0.03	0.04
query40	0.16	0.12	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 107.48 s
Total hot run time: 29.91 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 5, 2024

PR approved by anyone and no changes requested.

@924060929 924060929 force-pushed the forbid_hidden_columns branch from 10a629f to 097f5fc Compare August 5, 2024 07:38
@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41843 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 097f5fcc32f079272adb80e034b5d419fc517301, data reload: false

------ Round 1 ----------------------------------
q1	18160	4152	4281	4152
q2	2543	208	212	208
q3	11108	1289	1412	1289
q4	11166	807	919	807
q5	7888	2981	3015	2981
q6	228	139	142	139
q7	1049	602	620	602
q8	9439	1914	1933	1914
q9	8391	6598	6616	6598
q10	8689	3831	3843	3831
q11	425	252	265	252
q12	404	237	236	236
q13	17756	2919	2929	2919
q14	273	240	245	240
q15	520	477	483	477
q16	496	383	385	383
q17	956	914	900	900
q18	8034	7345	7476	7345
q19	1386	1206	1212	1206
q20	569	322	350	322
q21	5438	4779	4761	4761
q22	349	281	292	281
Total cold run time: 115267 ms
Total hot run time: 41843 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4066	4006	3997	3997
q2	326	221	226	221
q3	2974	3011	2972	2972
q4	1895	1837	1857	1837
q5	5251	5225	5206	5206
q6	215	130	130	130
q7	2059	1694	1705	1694
q8	3173	3244	3244	3244
q9	8276	8231	8267	8231
q10	3738	3857	3831	3831
q11	549	473	448	448
q12	725	570	552	552
q13	14231	2965	2932	2932
q14	288	259	254	254
q15	524	482	481	481
q16	449	400	403	400
q17	1744	1688	1683	1683
q18	7687	7326	7226	7226
q19	1657	1673	1658	1658
q20	1968	1761	1751	1751
q21	5375	5115	5206	5115
q22	527	463	445	445
Total cold run time: 67697 ms
Total hot run time: 54308 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 167550 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 097f5fcc32f079272adb80e034b5d419fc517301, data reload: false

query1	897	381	367	367
query2	6463	1669	1672	1669
query3	6659	213	227	213
query4	19367	17337	17097	17097
query5	4305	516	517	516
query6	277	181	173	173
query7	4604	297	289	289
query8	254	217	200	200
query9	8533	2390	2393	2390
query10	428	286	270	270
query11	10418	10070	10048	10048
query12	142	90	86	86
query13	1639	377	377	377
query14	9999	6127	6718	6127
query15	201	171	167	167
query16	7242	449	466	449
query17	987	568	557	557
query18	1909	293	286	286
query19	199	149	154	149
query20	93	93	87	87
query21	202	99	99	99
query22	4357	4095	4092	4092
query23	33776	33187	32926	32926
query24	10434	3113	3046	3046
query25	709	396	403	396
query26	1788	156	153	153
query27	2984	282	286	282
query28	6946	1994	1966	1966
query29	1294	425	431	425
query30	287	151	157	151
query31	966	748	759	748
query32	109	60	57	57
query33	783	311	315	311
query34	959	481	481	481
query35	878	752	723	723
query36	1002	870	877	870
query37	287	82	80	80
query38	2861	2771	2772	2771
query39	880	797	816	797
query40	286	122	110	110
query41	46	46	44	44
query42	131	99	104	99
query43	486	409	438	409
query44	1201	744	726	726
query45	210	182	177	177
query46	1147	836	794	794
query47	1799	1682	1697	1682
query48	360	291	294	291
query49	1192	443	427	427
query50	985	444	445	444
query51	6883	6644	6731	6644
query52	109	92	92	92
query53	267	190	183	183
query54	638	450	453	450
query55	77	79	78	78
query56	302	260	264	260
query57	1150	1029	1044	1029
query58	292	289	270	270
query59	2558	2414	2224	2224
query60	310	271	278	271
query61	122	91	94	91
query62	926	655	684	655
query63	220	195	193	193
query64	5839	1915	1880	1880
query65	3218	3111	3093	3093
query66	1454	335	348	335
query67	15219	14912	14685	14685
query68	4436	571	610	571
query69	458	303	296	296
query70	1118	1020	1154	1020
query71	405	283	291	283
query72	7137	2721	2475	2475
query73	817	332	327	327
query74	5970	5619	5616	5616
query75	3454	2737	2730	2730
query76	2531	1304	1411	1304
query77	463	335	320	320
query78	9494	8915	8805	8805
query79	2282	537	527	527
query80	1011	535	505	505
query81	566	230	233	230
query82	801	130	133	130
query83	259	178	165	165
query84	276	82	77	77
query85	1706	308	291	291
query86	496	313	311	311
query87	3269	3096	3055	3055
query88	3886	2413	2409	2409
query89	392	294	281	281
query90	1891	193	200	193
query91	126	101	103	101
query92	62	54	54	54
query93	2277	623	624	623
query94	917	324	295	295
query95	377	258	264	258
query96	659	283	276	276
query97	3219	3074	3028	3028
query98	217	199	195	195
query99	1733	1272	1302	1272
Total cold run time: 266250 ms
Total hot run time: 167550 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 097f5fcc32f079272adb80e034b5d419fc517301, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.05	0.04
query3	0.22	0.04	0.05
query4	1.68	0.09	0.07
query5	0.49	0.48	0.48
query6	1.13	0.71	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.59	0.52	0.52
query10	0.56	0.56	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.13
query13	0.62	0.60	0.60
query14	0.76	0.83	0.79
query15	0.89	0.86	0.87
query16	0.34	0.34	0.37
query17	1.00	0.99	1.00
query18	0.22	0.22	0.22
query19	1.84	1.75	1.70
query20	0.01	0.01	0.01
query21	15.40	0.76	0.66
query22	3.89	8.60	1.50
query23	18.08	1.33	1.31
query24	2.26	0.22	0.22
query25	0.19	0.09	0.08
query26	0.31	0.22	0.22
query27	0.45	0.24	0.22
query28	13.16	1.00	0.97
query29	12.63	3.31	3.27
query30	0.26	0.06	0.05
query31	2.88	0.41	0.40
query32	3.24	0.50	0.48
query33	2.89	2.99	2.97
query34	15.45	4.24	4.24
query35	4.30	4.27	4.30
query36	0.67	0.49	0.47
query37	0.18	0.16	0.15
query38	0.16	0.16	0.14
query39	0.04	0.03	0.04
query40	0.15	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 107.64 s
Total hot run time: 30.16 s

@924060929 924060929 merged commit 9eae4ba into apache:master Aug 6, 2024
30 checks passed
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 6, 2024
@924060929
[chore](sql) Forbid show hidden columns and create table with hidden …
fb38c41 
…column (apache#38796)

Forbid show hidden columns and create table with hidden column

(cherry picked from commit 9eae4ba)
This was referenced Aug 6, 2024
Closed
Merged
Merged
924060929 added a commit that referenced this pull request Aug 6, 2024
@924060929
[chore](sql) Forbid show hidden columns and create table with hidden …
fcb4483 
…column (#38796) (#38924)

Forbid show hidden columns and create table with hidden column
@924060929 924060929 mentioned this pull request Aug 7, 2024
Merged
@morrySnow morrySnow removed the doing label Aug 7, 2024
924060929 added a commit that referenced this pull request Aug 7, 2024
@924060929
[fix](test) fix conflict create table by different regression test (#…
bba78e6 
…39004)

fix conflict create table by different regression tests, introduced by
#38796
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 7, 2024
@924060929
[fix](test) fix conflict create table by different regression test (a…
ea50740 
…pache#39004)

fix conflict create table by different regression tests, introduced by
apache#38796

(cherry picked from commit bba78e6)
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 8, 2024
@924060929
[fix](test) fix conflict create table by different regression test (a…
8963258 
…pache#39004)

fix conflict create table by different regression tests, introduced by
apache#38796

(cherry picked from commit bba78e6)
924060929 added a commit that referenced this pull request Aug 8, 2024
@924060929
[fix](test) fix conflict create table by different regression test (#…
3291878 
…39004) (#39059)

fix conflict create table by different regression tests, introduced by #38796
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@924060929 @dataroaring
[chore](sql) Forbid show hidden columns and create table with hidden …
1669b1d 
…column (#38796)

Forbid show hidden columns and create table with hidden column
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@924060929 @dataroaring
[fix](test) fix conflict create table by different regression test (#…
b087d30 
…39004)

fix conflict create table by different regression tests, introduced by
#38796
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@924060929 @wyxxxcat
[fix](test) fix conflict create table by different regression test (a…
04b8312 
…pache#39004)

fix conflict create table by different regression tests, introduced by
apache#38796
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@924060929 @dataroaring
[chore](sql) Forbid show hidden columns and create table with hidden …
d706ee7 
…column (#38796)

Forbid show hidden columns and create table with hidden column
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@924060929 @dataroaring
[fix](test) fix conflict create table by different regression test (#…
a88a47a 
…39004)

fix conflict create table by different regression tests, introduced by
#38796
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow left review comments

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@starocean999 starocean999 starocean999 approved these changes

@feiniaofeiafei feiniaofeiafei feiniaofeiafei approved these changes

@xzj7019 xzj7019 xzj7019 approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

@morningman morningman Awaiting requested review from morningman morningman is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@924060929 @doris-robot @CalvinKirs @starocean999 @feiniaofeiafei @morrySnow @xzj7019 @dataroaring