Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bugfix](hudi)add timetravel for nereids #38324

Merged
merged 4 commits into from
Jul 30, 2024
Merged

[bugfix](hudi)add timetravel for nereids #38324

merged 4 commits into from
Jul 30, 2024

Conversation

wuwenchi
Copy link
Contributor

Proposed changes

  1. add timetravel for nereids.
select * from tb FOR TIME AS OF "2024-07-24 19:58:43";
select * from tb FOR TIME AS OF "20240724195843";
  1. Add ugi authentication to the thread pool

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39751 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eedf4cd664353b993dc2706d1ed83955cd282e60, data reload: false

------ Round 1 ----------------------------------
q1	17618	4354	4261	4261
q2	2014	187	185	185
q3	10464	1255	1075	1075
q4	10180	841	738	738
q5	7557	2701	2733	2701
q6	220	143	144	143
q7	968	596	600	596
q8	9224	2073	2061	2061
q9	8674	6557	6554	6554
q10	8824	3759	3757	3757
q11	462	245	234	234
q12	443	223	226	223
q13	17778	2957	3015	2957
q14	268	235	233	233
q15	518	471	483	471
q16	494	381	383	381
q17	949	652	690	652
q18	8054	7421	7427	7421
q19	6168	1363	1285	1285
q20	674	333	348	333
q21	4945	3205	3218	3205
q22	348	285	292	285
Total cold run time: 116844 ms
Total hot run time: 39751 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4932	4215	4212	4212
q2	376	275	252	252
q3	3016	2803	2908	2803
q4	1974	1714	1685	1685
q5	5653	5471	5494	5471
q6	222	132	137	132
q7	2196	1927	1852	1852
q8	3255	3425	3397	3397
q9	8775	8859	8928	8859
q10	4123	3847	3790	3790
q11	613	522	503	503
q12	810	666	693	666
q13	17041	3185	3193	3185
q14	315	297	306	297
q15	523	510	492	492
q16	500	426	456	426
q17	1836	1534	1511	1511
q18	8129	8041	7756	7756
q19	1749	1635	1464	1464
q20	2136	1889	1863	1863
q21	5320	4952	4805	4805
q22	640	501	506	501
Total cold run time: 74134 ms
Total hot run time: 55922 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173486 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eedf4cd664353b993dc2706d1ed83955cd282e60, data reload: false

query1	921	372	368	368
query2	6431	1892	1880	1880
query3	6639	211	218	211
query4	25604	17402	17351	17351
query5	3729	472	475	472
query6	285	174	173	173
query7	4598	302	299	299
query8	267	196	199	196
query9	8512	2414	2384	2384
query10	430	296	276	276
query11	12778	10198	10077	10077
query12	119	86	90	86
query13	1654	372	374	372
query14	9946	7396	7688	7396
query15	222	168	158	158
query16	7643	460	463	460
query17	1124	573	522	522
query18	1945	274	279	274
query19	194	154	151	151
query20	91	81	85	81
query21	209	129	125	125
query22	4289	4125	4034	4034
query23	34177	33727	33625	33625
query24	11143	2896	2939	2896
query25	588	385	393	385
query26	706	155	158	155
query27	2205	275	290	275
query28	6833	2076	2080	2076
query29	905	650	642	642
query30	247	152	151	151
query31	960	755	783	755
query32	99	56	61	56
query33	756	339	320	320
query34	951	512	504	504
query35	871	756	790	756
query36	1158	963	966	963
query37	143	82	85	82
query38	2958	2841	2838	2838
query39	926	810	814	810
query40	204	125	121	121
query41	46	44	44	44
query42	120	105	99	99
query43	498	480	451	451
query44	1215	725	723	723
query45	209	173	176	173
query46	1084	731	714	714
query47	1873	1764	1771	1764
query48	370	306	325	306
query49	827	414	423	414
query50	775	390	393	390
query51	6769	6753	6621	6621
query52	100	95	93	93
query53	363	293	280	280
query54	888	462	447	447
query55	73	74	74	74
query56	286	279	264	264
query57	1124	1017	1053	1017
query58	259	248	279	248
query59	2859	2788	2593	2593
query60	299	276	279	276
query61	94	92	97	92
query62	802	656	655	655
query63	315	296	289	289
query64	9540	2224	1671	1671
query65	3163	3076	3112	3076
query66	744	327	333	327
query67	15476	15064	15160	15064
query68	4490	560	556	556
query69	466	328	342	328
query70	1209	1068	1091	1068
query71	411	285	288	285
query72	7084	5681	5125	5125
query73	740	326	329	326
query74	6051	5685	5712	5685
query75	3413	2726	2711	2711
query76	2282	1034	937	937
query77	446	316	307	307
query78	9643	9850	8987	8987
query79	2011	526	524	524
query80	2482	479	539	479
query81	593	214	223	214
query82	970	133	141	133
query83	299	165	169	165
query84	268	93	88	88
query85	1134	304	313	304
query86	399	317	324	317
query87	3342	3104	3090	3090
query88	3388	2367	2372	2367
query89	472	395	386	386
query90	1729	193	198	193
query91	134	99	103	99
query92	61	49	50	49
query93	1462	516	519	516
query94	1084	290	294	290
query95	408	319	322	319
query96	589	275	269	269
query97	3250	3006	3037	3006
query98	220	202	194	194
query99	1666	1258	1248	1248
Total cold run time: 275827 ms
Total hot run time: 173486 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eedf4cd664353b993dc2706d1ed83955cd282e60, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.11	0.09
query5	0.51	0.48	0.47
query6	1.13	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.13	0.12
query13	0.61	0.58	0.59
query14	0.77	0.78	0.77
query15	0.85	0.81	0.81
query16	0.37	0.37	0.36
query17	1.04	0.95	0.94
query18	0.24	0.22	0.21
query19	1.80	1.75	1.65
query20	0.02	0.01	0.01
query21	15.40	0.75	0.66
query22	4.18	7.36	2.20
query23	18.29	1.45	1.33
query24	2.13	0.24	0.22
query25	0.15	0.09	0.09
query26	0.29	0.21	0.21
query27	0.45	0.23	0.24
query28	13.27	1.02	1.00
query29	12.59	3.37	3.34
query30	0.24	0.06	0.05
query31	2.85	0.39	0.38
query32	3.28	0.48	0.47
query33	2.91	2.84	2.88
query34	17.05	4.33	4.34
query35	4.42	4.43	4.43
query36	0.65	0.49	0.47
query37	0.19	0.16	0.15
query38	0.15	0.16	0.14
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 109.69 s
Total hot run time: 30.87 s

morningman
morningman approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

wsjz
wsjz approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@wsjz wsjz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

hubgeter
hubgeter approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@hubgeter hubgeter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 446d2a0 into apache:master Jul 30, 2024
31 of 33 checks passed
wuwenchi added a commit to wuwenchi/doris_new that referenced this pull request Jul 31, 2024
@wuwenchi
[bugfix](hudi)add timetravel for nereids (apache#38324)
a8d1a39 
## Proposed changes

1. add timetravel for nereids.

```
select * from tb FOR TIME AS OF "2024-07-24 19:58:43";
select * from tb FOR TIME AS OF "20240724195843";
```
2. Add ugi authentication to the thread pool

(cherry picked from commit 446d2a0)
@wuwenchi wuwenchi mentioned this pull request Jul 31, 2024
Merged
wuwenchi added a commit to wuwenchi/doris_new that referenced this pull request Jul 31, 2024
@wuwenchi
[bugfix](hudi)add timetravel for nereids (apache#38324)
3f135ca 
1. add timetravel for nereids.

```
select * from tb FOR TIME AS OF "2024-07-24 19:58:43";
select * from tb FOR TIME AS OF "20240724195843";
```
2. Add ugi authentication to the thread pool

(cherry picked from commit 446d2a0)
yiguolei pushed a commit that referenced this pull request Aug 1, 2024
@wuwenchi
[bugfix](hudi)add timetravel for nereids for 2.1 (#38324) (#38582)
057ee19 
## Proposed changes

bp #38324
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@wuwenchi @dataroaring
[bugfix](hudi)add timetravel for nereids (#38324)
092941c 
## Proposed changes

1. add timetravel for nereids.

```
select * from tb FOR TIME AS OF "2024-07-24 19:58:43";
select * from tb FOR TIME AS OF "20240724195843";
```
2. Add ugi authentication to the thread pool
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@wuwenchi @dataroaring
[bugfix](hudi)add timetravel for nereids (#38324)
675f8a1 
## Proposed changes

1. add timetravel for nereids.

```
select * from tb FOR TIME AS OF "2024-07-24 19:58:43";
select * from tb FOR TIME AS OF "20240724195843";
```
2. Add ugi authentication to the thread pool
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@wsjz wsjz wsjz approved these changes

@hubgeter hubgeter hubgeter approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@wuwenchi @doris-robot @morningman @wsjz @hubgeter @yiguolei @dataroaring