Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature](JsonReader) support sepecify $. as root column in json path #38213

Merged
merged 2 commits into from
Jul 29, 2024
Merged

[Feature](JsonReader) support sepecify $. as root column in json path #38213

merged 2 commits into from
Jul 29, 2024

Conversation

eldenmoon
Copy link
Member 

curl --location-trusted -u root:  -T value.json -H "read_json_by_line:true" -H "format:json"  -H "max_filter_ratio:0.5" -H "jsonpaths: [\"$.id\", \"$.entity_id\", \"$.\"]" http://127.0.0.1:8149/api/regression_test/records/_stream_load

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39598 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b9a7990770f60d182c7aeb186091aa497bc86ebd, data reload: false

------ Round 1 ----------------------------------
q1	17605	4311	4297	4297
q2	2014	190	184	184
q3	10445	1226	1115	1115
q4	10187	740	724	724
q5	7556	2693	2635	2635
q6	218	134	135	134
q7	943	587	593	587
q8	9214	2043	2066	2043
q9	8604	6522	6503	6503
q10	8807	3762	3810	3762
q11	438	237	235	235
q12	479	232	221	221
q13	17771	2981	3009	2981
q14	280	226	235	226
q15	524	484	485	484
q16	497	378	379	378
q17	959	716	656	656
q18	7968	7335	7514	7335
q19	4166	1380	1425	1380
q20	698	316	331	316
q21	4889	3200	3118	3118
q22	355	284	285	284
Total cold run time: 114617 ms
Total hot run time: 39598 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4345	4257	4201	4201
q2	377	285	279	279
q3	3007	2734	2813	2734
q4	1932	1726	1740	1726
q5	5549	5615	5473	5473
q6	221	128	133	128
q7	2218	1846	1771	1771
q8	3325	3405	3387	3387
q9	8777	8797	8843	8797
q10	4112	3887	3856	3856
q11	585	483	495	483
q12	789	649	676	649
q13	16221	3174	3147	3147
q14	316	296	287	287
q15	523	510	486	486
q16	474	443	448	443
q17	1822	1482	1535	1482
q18	7944	8089	7990	7990
q19	2479	1658	1603	1603
q20	2211	1916	1869	1869
q21	5091	4957	4810	4810
q22	575	489	514	489
Total cold run time: 72893 ms
Total hot run time: 56090 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174941 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9a7990770f60d182c7aeb186091aa497bc86ebd, data reload: false

query1	908	370	368	368
query2	6433	1979	1932	1932
query3	6638	207	223	207
query4	24827	17609	17511	17511
query5	3688	490	471	471
query6	272	184	177	177
query7	4586	284	292	284
query8	243	204	194	194
query9	8525	2471	2463	2463
query10	449	277	294	277
query11	11443	10106	10113	10106
query12	125	87	83	83
query13	1640	379	380	379
query14	9531	7688	7605	7605
query15	228	163	168	163
query16	7761	513	480	480
query17	1612	580	540	540
query18	1945	291	285	285
query19	201	162	159	159
query20	93	82	87	82
query21	210	129	128	128
query22	4355	4046	4100	4046
query23	34013	34084	33812	33812
query24	10647	2942	2936	2936
query25	603	423	419	419
query26	719	158	156	156
query27	2189	283	287	283
query28	6471	2102	2071	2071
query29	902	660	656	656
query30	249	161	154	154
query31	998	766	781	766
query32	96	55	60	55
query33	692	371	360	360
query34	927	510	511	510
query35	908	769	777	769
query36	1144	990	1001	990
query37	139	79	81	79
query38	2988	2811	2760	2760
query39	878	796	803	796
query40	207	126	119	119
query41	46	43	44	43
query42	118	97	96	96
query43	500	473	474	473
query44	1076	722	709	709
query45	193	162	164	162
query46	1079	698	692	692
query47	1850	1746	1756	1746
query48	384	291	299	291
query49	836	423	414	414
query50	789	387	394	387
query51	6764	6721	6619	6619
query52	103	90	91	90
query53	358	293	286	286
query54	899	462	451	451
query55	75	73	74	73
query56	292	282	267	267
query57	1125	1034	1025	1025
query58	278	248	262	248
query59	3034	2555	2689	2555
query60	318	279	308	279
query61	95	93	100	93
query62	786	649	649	649
query63	310	293	288	288
query64	9089	2246	7448	2246
query65	3126	3112	3123	3112
query66	750	331	354	331
query67	15565	14913	14824	14824
query68	6517	533	545	533
query69	731	444	392	392
query70	1195	1098	1142	1098
query71	488	274	285	274
query72	8924	5608	5650	5608
query73	770	327	326	326
query74	5978	5599	5690	5599
query75	4839	2659	2741	2659
query76	4488	940	908	908
query77	831	318	311	311
query78	9829	8862	8915	8862
query79	8484	509	522	509
query80	1078	488	480	480
query81	586	223	218	218
query82	780	133	136	133
query83	337	164	168	164
query84	278	86	87	86
query85	1323	319	312	312
query86	453	291	317	291
query87	3287	3107	3168	3107
query88	4814	2511	2469	2469
query89	609	401	381	381
query90	1923	194	195	194
query91	133	99	98	98
query92	64	50	51	50
query93	6786	498	501	498
query94	1244	304	287	287
query95	408	321	324	321
query96	611	273	285	273
query97	3158	2982	3039	2982
query98	223	195	192	192
query99	1531	1246	1255	1246
Total cold run time: 293021 ms
Total hot run time: 174941 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b9a7990770f60d182c7aeb186091aa497bc86ebd, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.08	0.10
query5	0.50	0.50	0.48
query6	1.14	0.71	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.55	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.13	0.13
query13	0.59	0.59	0.59
query14	0.77	0.77	0.78
query15	0.84	0.80	0.82
query16	0.37	0.36	0.38
query17	1.03	1.03	0.96
query18	0.23	0.22	0.22
query19	1.86	1.67	1.77
query20	0.01	0.01	0.01
query21	15.42	0.84	0.67
query22	4.32	7.06	1.94
query23	18.26	1.36	1.31
query24	2.09	0.23	0.23
query25	0.16	0.10	0.09
query26	0.31	0.22	0.20
query27	0.46	0.23	0.23
query28	13.32	1.03	1.00
query29	12.64	3.26	3.23
query30	0.26	0.07	0.05
query31	2.88	0.38	0.39
query32	3.25	0.48	0.48
query33	2.91	2.84	2.90
query34	16.94	4.35	4.33
query35	4.45	4.43	4.38
query36	0.65	0.48	0.46
query37	0.18	0.16	0.15
query38	0.15	0.14	0.15
query39	0.05	0.03	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.06
query43	0.04	0.04	0.04
Total cold run time: 109.89 s
Total hot run time: 30.5 s

@eldenmoon
Copy link
Member Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40097 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0035e09c1bd2f6cf653c49f2061f923ce7e3fb5, data reload: false

------ Round 1 ----------------------------------
q1	17977	4422	4342	4342
q2	2186	192	187	187
q3	10477	1161	1177	1161
q4	10193	753	798	753
q5	7537	2784	2782	2782
q6	223	140	137	137
q7	966	600	593	593
q8	9381	2094	2076	2076
q9	8567	6511	6517	6511
q10	8748	3794	3771	3771
q11	453	245	238	238
q12	391	223	216	216
q13	18177	2978	2989	2978
q14	300	241	238	238
q15	525	482	492	482
q16	473	397	381	381
q17	978	625	678	625
q18	8072	7491	7451	7451
q19	8470	1314	1292	1292
q20	685	327	323	323
q21	4926	3276	3345	3276
q22	346	294	284	284
Total cold run time: 120051 ms
Total hot run time: 40097 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4376	4309	4247	4247
q2	372	273	267	267
q3	3017	2780	2772	2772
q4	1877	1607	1532	1532
q5	5306	5292	5341	5292
q6	217	130	131	130
q7	2168	1757	1742	1742
q8	3198	3368	3309	3309
q9	8441	8369	8422	8369
q10	3926	3762	3679	3679
q11	601	482	473	473
q12	801	611	635	611
q13	17174	2978	2983	2978
q14	289	275	276	275
q15	520	477	490	477
q16	481	410	418	410
q17	1773	1484	1456	1456
q18	7621	7526	7404	7404
q19	1646	1449	1533	1449
q20	1994	1785	1774	1774
q21	4881	4672	4690	4672
q22	591	479	493	479
Total cold run time: 71270 ms
Total hot run time: 53797 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172820 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0035e09c1bd2f6cf653c49f2061f923ce7e3fb5, data reload: false

query1	910	362	365	362
query2	6431	1899	1802	1802
query3	6675	208	224	208
query4	28501	17383	17284	17284
query5	4176	464	470	464
query6	276	180	155	155
query7	4586	286	288	286
query8	239	203	182	182
query9	8517	2462	2436	2436
query10	455	287	275	275
query11	11552	10138	10146	10138
query12	140	88	83	83
query13	1650	372	365	365
query14	10230	7805	7774	7774
query15	236	167	161	161
query16	7537	496	428	428
query17	1577	522	533	522
query18	1343	269	276	269
query19	196	147	146	146
query20	90	103	82	82
query21	212	134	131	131
query22	4236	4012	3937	3937
query23	33586	33191	33170	33170
query24	12061	2924	2940	2924
query25	677	374	392	374
query26	1796	146	148	146
query27	2855	274	274	274
query28	7390	1981	1971	1971
query29	1043	605	608	605
query30	283	147	146	146
query31	935	725	739	725
query32	94	54	51	51
query33	769	338	327	327
query34	895	482	496	482
query35	844	723	743	723
query36	1121	957	925	925
query37	268	78	79	78
query38	2865	2775	2747	2747
query39	869	806	812	806
query40	269	118	119	118
query41	48	42	43	42
query42	111	96	100	96
query43	508	482	462	462
query44	1208	710	714	710
query45	188	161	162	161
query46	1092	721	702	702
query47	1897	1788	1797	1788
query48	365	298	295	295
query49	1198	405	430	405
query50	779	404	407	404
query51	6793	6698	6637	6637
query52	101	97	94	94
query53	358	284	286	284
query54	1059	441	445	441
query55	75	75	72	72
query56	288	268	272	268
query57	1128	1035	1019	1019
query58	253	256	255	255
query59	2834	2534	2642	2534
query60	301	277	296	277
query61	96	95	97	95
query62	833	688	661	661
query63	324	291	289	289
query64	10490	2226	1648	1648
query65	3153	3123	3117	3117
query66	1276	325	332	325
query67	15513	14881	14962	14881
query68	4502	536	539	536
query69	480	339	343	339
query70	1219	1162	1130	1130
query71	449	276	281	276
query72	7039	6066	5365	5365
query73	751	338	337	337
query74	6073	5674	5749	5674
query75	3454	2714	2701	2701
query76	2826	882	878	878
query77	487	298	292	292
query78	9553	8993	9000	8993
query79	2694	525	506	506
query80	2113	463	459	459
query81	594	223	222	222
query82	891	147	134	134
query83	290	164	165	164
query84	288	95	85	85
query85	2076	320	302	302
query86	492	337	305	305
query87	3366	3118	3085	3085
query88	3984	2458	2455	2455
query89	481	383	377	377
query90	1856	203	196	196
query91	143	111	116	111
query92	153	49	53	49
query93	3269	495	491	491
query94	1280	272	268	268
query95	413	313	316	313
query96	621	281	272	272
query97	3199	3014	3013	3013
query98	229	200	193	193
query99	1694	1255	1268	1255
Total cold run time: 287613 ms
Total hot run time: 172820 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0035e09c1bd2f6cf653c49f2061f923ce7e3fb5, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.09	0.10
query5	0.49	0.48	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.51
query10	0.55	0.53	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.75	0.77	0.81
query15	0.86	0.80	0.81
query16	0.36	0.37	0.36
query17	0.97	0.97	1.00
query18	0.22	0.22	0.21
query19	1.75	1.69	1.70
query20	0.01	0.01	0.01
query21	15.42	0.77	0.67
query22	4.25	7.61	1.73
query23	18.26	1.43	1.38
query24	2.07	0.26	0.23
query25	0.16	0.09	0.08
query26	0.30	0.21	0.22
query27	0.46	0.23	0.23
query28	13.21	1.01	1.02
query29	12.68	3.30	3.26
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.28	0.48	0.47
query33	2.92	2.94	2.91
query34	16.97	4.36	4.36
query35	4.41	4.46	4.46
query36	0.65	0.48	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.12
query41	0.10	0.05	0.04
query42	0.05	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.51 s
Total hot run time: 30.61 s

@eldenmoon
[Feature](JsonReader) support sepecify $. as root column in json path
5dfec85 
```
curl --location-trusted -u root:  -T value.json -H "read_json_by_line:true" -H "format:json"  -H "max_filter_ratio:0.5" -H "jsonpaths: [\"$.id\", \"$.entity_id\", \"$.\"]" http://127.0.0.1:8149/api/regression_test/records/_stream_load
```
@eldenmoon
Copy link
Member Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39634 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5dfec853594460dcff83227b1323d58cc6e1de7e, data reload: false

------ Round 1 ----------------------------------
q1	17732	4393	4252	4252
q2	2005	203	184	184
q3	10455	1156	1123	1123
q4	10179	820	839	820
q5	7550	2701	2769	2701
q6	228	136	134	134
q7	955	593	603	593
q8	9210	2061	2075	2061
q9	8909	6568	6537	6537
q10	8803	3754	3784	3754
q11	447	233	235	233
q12	413	230	217	217
q13	17776	2984	2969	2969
q14	278	234	242	234
q15	528	483	480	480
q16	495	383	387	383
q17	966	569	681	569
q18	8033	7415	7387	7387
q19	7868	1407	1304	1304
q20	663	330	326	326
q21	4875	3096	3160	3096
q22	350	285	277	277
Total cold run time: 118718 ms
Total hot run time: 39634 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4445	4248	4227	4227
q2	381	273	261	261
q3	3001	2884	2825	2825
q4	2024	1676	1737	1676
q5	5627	5536	5481	5481
q6	223	129	139	129
q7	2194	1905	1812	1812
q8	3263	3377	3414	3377
q9	8817	8713	8879	8713
q10	4108	3884	3746	3746
q11	591	508	506	506
q12	822	654	656	654
q13	16340	3175	3208	3175
q14	340	285	293	285
q15	527	488	498	488
q16	502	467	444	444
q17	1810	1536	1492	1492
q18	8066	8118	7921	7921
q19	1782	1694	1475	1475
q20	2258	1868	1824	1824
q21	8063	4849	4822	4822
q22	593	515	512	512
Total cold run time: 75777 ms
Total hot run time: 55845 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174042 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5dfec853594460dcff83227b1323d58cc6e1de7e, data reload: false

query1	921	376	374	374
query2	6441	1900	1873	1873
query3	6668	207	214	207
query4	28304	17407	17367	17367
query5	3768	481	486	481
query6	267	182	169	169
query7	4578	297	284	284
query8	230	194	198	194
query9	8466	2423	2388	2388
query10	444	285	270	270
query11	11836	10102	10139	10102
query12	117	91	90	90
query13	1660	370	365	365
query14	10028	7059	7672	7059
query15	214	171	167	167
query16	7778	476	475	475
query17	1148	572	537	537
query18	1957	284	298	284
query19	195	152	153	152
query20	90	84	85	84
query21	210	133	126	126
query22	4571	4114	3952	3952
query23	34150	33697	33728	33697
query24	11137	3003	2943	2943
query25	672	412	413	412
query26	1223	160	156	156
query27	2389	269	277	269
query28	7192	2050	2048	2048
query29	907	689	645	645
query30	251	156	161	156
query31	958	801	773	773
query32	103	54	57	54
query33	762	343	345	343
query34	916	496	510	496
query35	863	761	797	761
query36	1206	964	966	964
query37	145	83	88	83
query38	3007	2840	2892	2840
query39	905	869	846	846
query40	218	119	123	119
query41	45	43	46	43
query42	118	99	99	99
query43	503	460	468	460
query44	1207	738	728	728
query45	195	161	157	157
query46	1091	767	723	723
query47	1867	1765	1764	1764
query48	364	278	289	278
query49	845	414	408	408
query50	789	404	383	383
query51	6752	6751	6627	6627
query52	108	93	90	90
query53	359	293	292	292
query54	935	446	435	435
query55	75	72	73	72
query56	281	261	261	261
query57	1114	1060	1068	1060
query58	257	299	274	274
query59	2807	2542	2592	2542
query60	305	273	297	273
query61	96	95	91	91
query62	818	663	659	659
query63	317	285	281	281
query64	9334	2213	1696	1696
query65	3193	3126	3151	3126
query66	758	332	329	329
query67	15850	15046	15025	15025
query68	8028	527	527	527
query69	739	454	381	381
query70	1230	1126	1063	1063
query71	524	281	278	278
query72	8811	5774	5931	5774
query73	906	324	320	320
query74	6216	5728	5777	5728
query75	4937	2714	2687	2687
query76	4873	907	966	907
query77	787	303	298	298
query78	9546	9093	16326	9093
query79	6078	515	505	505
query80	925	481	472	472
query81	580	222	224	222
query82	284	139	130	130
query83	303	171	166	166
query84	274	89	83	83
query85	1371	319	347	319
query86	448	297	313	297
query87	3308	3126	3133	3126
query88	3730	2361	2375	2361
query89	469	392	382	382
query90	1966	190	188	188
query91	129	99	101	99
query92	64	49	48	48
query93	1010	490	482	482
query94	1201	289	289	289
query95	403	311	315	311
query96	588	266	274	266
query97	3229	3044	3016	3016
query98	216	199	195	195
query99	1555	1251	1248	1248
Total cold run time: 292006 ms
Total hot run time: 174042 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5dfec853594460dcff83227b1323d58cc6e1de7e, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.05	0.04
query3	0.23	0.05	0.04
query4	1.67	0.07	0.08
query5	0.47	0.48	0.48
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.54	0.55
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.57
query14	0.76	0.77	0.79
query15	0.85	0.81	0.82
query16	0.37	0.35	0.35
query17	0.97	1.01	0.95
query18	0.22	0.22	0.22
query19	1.81	1.71	1.70
query20	0.01	0.01	0.01
query21	15.42	0.76	0.64
query22	4.41	7.10	1.69
query23	18.38	1.44	1.27
query24	2.07	0.24	0.23
query25	0.16	0.09	0.09
query26	0.30	0.21	0.21
query27	0.45	0.23	0.23
query28	13.36	1.04	1.01
query29	12.93	3.29	3.28
query30	0.26	0.06	0.05
query31	2.87	0.40	0.38
query32	3.24	0.48	0.47
query33	2.93	2.90	2.89
query34	17.12	4.34	4.35
query35	4.40	4.43	4.44
query36	0.65	0.46	0.47
query37	0.18	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.36 s
Total hot run time: 30.32 s

amorynan
amorynan approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@amorynan amorynan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@eldenmoon
Copy link
Member Author

run buildall

@eldenmoon
Copy link
Member Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39078 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0db4ed20219cf6f5d1a6310a587937c2acfcd36, data reload: false

------ Round 1 ----------------------------------
q1	17624	4811	4285	4285
q2	2006	198	189	189
q3	10429	1154	1070	1070
q4	10141	693	709	693
q5	7539	2764	2689	2689
q6	223	136	137	136
q7	939	591	581	581
q8	9214	1920	1881	1881
q9	8841	6498	6528	6498
q10	8687	3728	3767	3728
q11	488	244	245	244
q12	483	215	216	215
q13	19023	3001	2998	2998
q14	292	234	238	234
q15	533	484	494	484
q16	508	391	380	380
q17	979	647	742	647
q18	8048	7447	7424	7424
q19	4733	940	1032	940
q20	689	334	331	331
q21	5060	3135	3160	3135
q22	347	298	296	296
Total cold run time: 116826 ms
Total hot run time: 39078 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4413	4258	4334	4258
q2	388	269	268	268
q3	2956	2784	2885	2784
q4	1945	1765	1673	1673
q5	5640	5557	5516	5516
q6	220	131	130	130
q7	2180	1819	1877	1819
q8	3244	3462	3399	3399
q9	8778	8792	8927	8792
q10	4099	3863	3726	3726
q11	589	505	510	505
q12	803	652	653	652
q13	17013	3158	3156	3156
q14	311	289	282	282
q15	526	486	489	486
q16	486	453	440	440
q17	1826	1521	1517	1517
q18	7940	8087	7909	7909
q19	2221	1433	1465	1433
q20	2405	1879	1851	1851
q21	7606	4808	4697	4697
q22	636	513	564	513
Total cold run time: 76225 ms
Total hot run time: 55806 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171702 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0db4ed20219cf6f5d1a6310a587937c2acfcd36, data reload: false

query1	906	367	363	363
query2	6436	1871	1774	1774
query3	6639	206	218	206
query4	28296	17604	17313	17313
query5	3681	482	473	473
query6	269	169	180	169
query7	4577	279	286	279
query8	240	193	192	192
query9	8414	2443	2420	2420
query10	440	277	254	254
query11	11512	9971	10141	9971
query12	117	86	84	84
query13	1630	363	371	363
query14	10227	7059	7793	7059
query15	227	195	163	163
query16	7089	448	509	448
query17	1481	562	535	535
query18	1675	277	275	275
query19	205	137	142	137
query20	86	93	83	83
query21	210	99	97	97
query22	4267	3998	3776	3776
query23	33913	33586	33461	33461
query24	10908	2925	2957	2925
query25	588	416	408	408
query26	724	153	154	153
query27	2231	271	287	271
query28	5755	2095	2074	2074
query29	725	454	447	447
query30	261	152	153	152
query31	992	768	751	751
query32	97	59	55	55
query33	716	332	318	318
query34	871	482	480	480
query35	888	744	722	722
query36	1131	950	951	950
query37	147	85	81	81
query38	2940	2806	2879	2806
query39	867	857	855	855
query40	202	147	114	114
query41	54	46	44	44
query42	111	97	100	97
query43	492	448	475	448
query44	1201	747	722	722
query45	206	174	176	174
query46	1088	731	727	727
query47	1826	1742	1786	1742
query48	362	290	293	290
query49	830	418	413	413
query50	789	396	396	396
query51	6892	6671	6689	6671
query52	97	90	87	87
query53	249	184	179	179
query54	870	443	427	427
query55	75	72	76	72
query56	291	282	274	274
query57	1099	1043	1064	1043
query58	265	282	311	282
query59	2781	2556	2645	2556
query60	338	296	324	296
query61	94	95	94	94
query62	779	625	652	625
query63	202	179	179	179
query64	9223	2255	1711	1711
query65	3199	3065	3099	3065
query66	772	331	330	330
query67	15286	14886	14833	14833
query68	4488	552	553	552
query69	446	296	293	293
query70	1098	1090	1100	1090
query71	362	273	262	262
query72	7000	6002	5692	5692
query73	753	327	329	327
query74	6075	5551	5667	5551
query75	3361	2659	2619	2619
query76	2100	952	872	872
query77	418	298	308	298
query78	9675	8969	8805	8805
query79	3031	529	529	529
query80	2140	490	489	489
query81	612	215	217	215
query82	783	132	134	132
query83	309	182	173	173
query84	261	85	76	76
query85	1556	325	355	325
query86	468	316	303	303
query87	3231	3113	3046	3046
query88	3855	2380	2391	2380
query89	397	285	278	278
query90	1694	189	182	182
query91	128	97	106	97
query92	63	48	49	48
query93	2643	545	550	545
query94	812	284	291	284
query95	345	259	260	259
query96	587	280	264	264
query97	3213	3082	3043	3043
query98	241	207	192	192
query99	1571	1247	1268	1247
Total cold run time: 275767 ms
Total hot run time: 171702 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0db4ed20219cf6f5d1a6310a587937c2acfcd36, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.67	0.07	0.07
query5	0.49	0.50	0.48
query6	1.13	0.72	0.74
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.54	0.50	0.48
query10	0.55	0.55	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.77	0.78	0.81
query15	0.86	0.82	0.81
query16	0.37	0.36	0.37
query17	1.02	1.03	1.02
query18	0.24	0.22	0.23
query19	1.84	1.73	1.77
query20	0.01	0.01	0.01
query21	15.41	0.78	0.67
query22	4.52	7.00	1.74
query23	18.32	1.48	1.26
query24	2.10	0.23	0.24
query25	0.16	0.09	0.09
query26	0.29	0.21	0.21
query27	0.46	0.24	0.23
query28	13.20	1.03	0.99
query29	12.68	3.31	3.29
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.30	0.49	0.46
query33	2.92	2.91	2.91
query34	16.97	4.32	4.35
query35	4.39	4.37	4.46
query36	0.65	0.48	0.46
query37	0.19	0.15	0.14
query38	0.15	0.15	0.15
query39	0.04	0.03	0.03
query40	0.15	0.13	0.12
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.04 s
Total hot run time: 30.43 s

xiaokang
xiaokang approved these changes Jul 29, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang added the doing label Jul 29, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@xiaokang xiaokang merged commit 224e25e into apache:master Jul 29, 2024
26 of 28 checks passed
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 30, 2024
@eldenmoon
[Feature](JsonReader) support sepecify $. as root column in json pa…
ab04b5b 
…th (apache#38213)

```
curl --location-trusted -u root:  -T value.json -H "read_json_by_line:true" -H "format:json"  -H "max_filter_ratio:0.5" -H "jsonpaths: [\"$.id\", \"$.entity_id\", \"$.\"]" http://127.0.0.1:8149/api/regression_test/records/_stream_load
```
@eldenmoon eldenmoon mentioned this pull request Jul 30, 2024
Merged
eldenmoon added a commit that referenced this pull request Jul 30, 2024
@eldenmoon
[Pick](Variant) pick some fix (#38513)
70c308c 
#38318 
#38364 
#38291 
#38213 
#38413
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Aug 13, 2024
@eldenmoon
[Feature](JsonReader) support sepecify $. as root column in json pa…
7681503 
…th (apache#38213)

```
curl --location-trusted -u root:  -T value.json -H "read_json_by_line:true" -H "format:json"  -H "max_filter_ratio:0.5" -H "jsonpaths: [\"$.id\", \"$.entity_id\", \"$.\"]" http://127.0.0.1:8149/api/regression_test/records/_stream_load
```
@eldenmoon eldenmoon mentioned this pull request Aug 13, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 13, 2024
@eldenmoon
[Pick](Branch-2.1) pick json reader fix and support specify $. as col…
677435c 
…umn (#39271)

#39206
#38213
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaokang xiaokang xiaokang approved these changes

@amorynan amorynan amorynan approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@eldenmoon @doris-robot @xiaokang @amorynan @yiguolei