Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[branch-2.0][improvement](sqlserver catalog) Can configure whether to use encryption when connecting to SQL Server using the catalog #36940

Merged
merged 1 commit into from
Jul 1, 2024
Merged

[branch-2.0][improvement](sqlserver catalog) Can configure whether to use encryption when connecting to SQL Server using the catalog #36940

merged 1 commit into from
Jul 1, 2024

Conversation

zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Jun 27, 2024 •
edited
Loading

pick #36659
pick #37015
In previous versions, we used druid as the default JDBC connection pool, which can use custom decryption to parse the certificate when SQL Server encryption is turned on. However, in the new version, after changing HikariCP as the default connection pool, the SQLServer certificate cannot be parsed, so encryption needs to be turned off for normal use. Therefore, a parameter is added to decide whether to disable SQLServer encryption. It is not disabled by default.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zy-kkk
Copy link
Member Author

zy-kkk commented Jun 27, 2024

run buildall

@zy-kkk zy-kkk changed the title [improvement](sqlserver catalog) Can configure whether to use encryption when connecting to SQL Server using the catalog [branch-2.0][improvement](sqlserver catalog) Can configure whether to use encryption when connecting to SQL Server using the catalog Jun 27, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 50271 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40c086c77ab861c7b2408586f8d978582facaf39, data reload: false

------ Round 1 ----------------------------------
q1	17640	4405	4543	4405
q2	2091	163	145	145
q3	10263	1938	1960	1938
q4	10261	1263	1326	1263
q5	8610	3948	3980	3948
q6	233	124	123	123
q7	2020	1615	1619	1615
q8	9504	2762	2714	2714
q9	14069	10482	10497	10482
q10	8688	3539	3527	3527
q11	419	239	253	239
q12	475	300	311	300
q13	18397	3967	3991	3967
q14	359	339	327	327
q15	507	471	459	459
q16	662	569	568	568
q17	1132	933	929	929
q18	7240	6900	6883	6883
q19	1760	1681	1596	1596
q20	560	307	311	307
q21	4392	4106	4118	4106
q22	527	430	444	430
Total cold run time: 119809 ms
Total hot run time: 50271 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4324	4318	4314	4314
q2	329	224	225	224
q3	4182	4183	4135	4135
q4	2750	2763	2746	2746
q5	7170	7099	7116	7099
q6	239	118	120	118
q7	3273	2848	2932	2848
q8	4364	4483	4489	4483
q9	17404	17212	17119	17119
q10	4265	4264	4284	4264
q11	787	724	697	697
q12	1040	858	858	858
q13	6977	3742	3752	3742
q14	458	421	438	421
q15	521	464	457	457
q16	740	678	679	678
q17	3849	3902	3873	3873
q18	8838	8810	8800	8800
q19	1719	1643	1674	1643
q20	2356	2156	2105	2105
q21	8412	8474	8540	8474
q22	1059	965	958	958
Total cold run time: 85056 ms
Total hot run time: 80056 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204498 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40c086c77ab861c7b2408586f8d978582facaf39, data reload: false

query1	951	420	374	374
query2	6529	2729	2701	2701
query3	6915	207	208	207
query4	21257	18042	18023	18023
query5	19733	6519	6490	6490
query6	301	229	227	227
query7	4175	305	316	305
query8	404	460	392	392
query9	3092	2703	2615	2615
query10	426	298	303	298
query11	11276	10692	10683	10683
query12	124	75	72	72
query13	5607	693	684	684
query14	17771	13517	13237	13237
query15	401	242	254	242
query16	6462	287	263	263
query17	1693	1442	880	880
query18	2324	412	409	409
query19	211	154	162	154
query20	76	77	90	77
query21	192	97	93	93
query22	5117	5092	4974	4974
query23	32442	32181	32021	32021
query24	6889	6484	6520	6484
query25	535	438	416	416
query26	533	173	160	160
query27	1876	295	293	293
query28	6175	2367	2329	2329
query29	2912	2867	2741	2741
query30	244	168	168	168
query31	921	765	741	741
query32	74	65	60	60
query33	391	265	258	258
query34	856	462	463	462
query35	1131	959	945	945
query36	1257	1129	1255	1129
query37	97	58	64	58
query38	3107	2898	2892	2892
query39	1381	1329	1330	1329
query40	209	89	100	89
query41	47	43	46	43
query42	94	88	79	79
query43	746	651	762	651
query44	1142	723	719	719
query45	252	239	238	238
query46	1209	969	994	969
query47	1952	1884	1819	1819
query48	1010	735	727	727
query49	636	368	375	368
query50	857	605	611	605
query51	4737	4639	4646	4639
query52	102	77	93	77
query53	446	325	331	325
query54	2662	2431	2423	2423
query55	102	88	85	85
query56	240	236	218	218
query57	1187	1066	1175	1066
query58	216	206	206	206
query59	4134	4237	3898	3898
query60	223	200	198	198
query61	98	97	102	97
query62	908	472	459	459
query63	481	343	333	333
query64	2376	1520	1452	1452
query65	3626	3538	3553	3538
query66	798	377	388	377
query67	15487	15577	15796	15577
query68	9557	666	663	663
query69	574	344	341	341
query70	1708	1480	1343	1343
query71	413	304	298	298
query72	6521	3522	3509	3509
query73	732	322	316	316
query74	6306	5856	5951	5856
query75	5484	3716	3665	3665
query76	6049	1189	1200	1189
query77	1034	259	252	252
query78	12654	12027	12237	12027
query79	10972	666	666	666
query80	883	397	400	397
query81	489	245	232	232
query82	1173	98	97	97
query83	178	134	140	134
query84	264	71	69	69
query85	835	322	324	322
query86	339	291	297	291
query87	3204	3021	2974	2974
query88	4855	2337	2322	2322
query89	442	308	303	303
query90	1932	215	217	215
query91	173	146	139	139
query92	60	54	54	54
query93	5749	630	565	565
query94	665	215	206	206
query95	1119	1076	1076	1076
query96	654	330	328	328
query97	6553	6414	6430	6414
query98	190	177	180	177
query99	3028	897	921	897
Total cold run time: 316624 ms
Total hot run time: 204498 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40c086c77ab861c7b2408586f8d978582facaf39, data reload: false

query1	0.02	0.02	0.02
query2	0.08	0.02	0.03
query3	0.25	0.05	0.04
query4	1.78	0.07	0.07
query5	0.54	0.52	0.52
query6	1.23	0.62	0.62
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.53	0.49	0.49
query10	0.52	0.53	0.54
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.60	0.61
query14	0.78	0.80	0.77
query15	0.78	0.75	0.76
query16	0.37	0.40	0.36
query17	1.02	1.01	0.99
query18	0.20	0.29	0.26
query19	1.85	1.82	1.85
query20	0.01	0.00	0.01
query21	15.46	0.56	0.57
query22	2.02	2.13	1.75
query23	17.33	1.05	0.88
query24	5.37	1.09	1.12
query25	0.37	0.07	0.05
query26	0.65	0.16	0.16
query27	0.05	0.04	0.04
query28	7.51	0.72	0.71
query29	12.73	2.35	2.31
query30	0.61	0.54	0.55
query31	2.81	0.39	0.38
query32	3.35	0.49	0.49
query33	3.06	3.08	3.05
query34	15.25	4.82	4.80
query35	4.93	4.83	4.85
query36	1.05	1.03	1.02
query37	0.05	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.01
Total cold run time: 103.78 s
Total hot run time: 30.96 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 40c086c77ab861c7b2408586f8d978582facaf39 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@zy-kkk
[improvement](sqlserver catalog) Can configure whether to use encrypt…
7d78c3e 
…ion when connecting to SQL Server using the catalog
@zy-kkk zy-kkk force-pushed the sqlserver_encrypt_false_20 branch from 40c086c to 7d78c3e Compare June 28, 2024 12:04
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50102 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d78c3ef1aa9789e28a5383df3d6c877b1bff75e, data reload: false

------ Round 1 ----------------------------------
q1	17870	4432	4410	4410
q2	2088	161	150	150
q3	10471	1875	1925	1875
q4	10348	1208	1316	1208
q5	8645	3962	3914	3914
q6	233	149	127	127
q7	2042	1611	1609	1609
q8	9525	2711	2715	2711
q9	14324	10458	10613	10458
q10	8603	3519	3520	3519
q11	425	243	246	243
q12	470	308	305	305
q13	18352	3979	4070	3979
q14	352	338	330	330
q15	500	456	458	456
q16	661	573	578	573
q17	1136	953	935	935
q18	7233	6883	6893	6883
q19	1768	1668	1598	1598
q20	545	300	293	293
q21	4375	4086	4080	4080
q22	534	450	446	446
Total cold run time: 120500 ms
Total hot run time: 50102 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4308	4304	4277	4277
q2	319	221	234	221
q3	4193	4149	4120	4120
q4	2758	2763	2732	2732
q5	7204	7162	7085	7085
q6	237	121	119	119
q7	3176	2855	2822	2822
q8	4334	4474	4464	4464
q9	17363	17010	17111	17010
q10	4240	4268	4294	4268
q11	754	688	686	686
q12	1046	839	856	839
q13	6448	3702	3695	3695
q14	470	436	419	419
q15	513	458	458	458
q16	724	692	693	692
q17	3833	3858	3834	3834
q18	8748	8819	8665	8665
q19	1734	1716	1655	1655
q20	2367	2099	2093	2093
q21	8367	8453	8376	8376
q22	1057	965	934	934
Total cold run time: 84193 ms
Total hot run time: 79464 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203469 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d78c3ef1aa9789e28a5383df3d6c877b1bff75e, data reload: false

query1	930	424	377	377
query2	6524	2692	2574	2574
query3	6929	201	205	201
query4	20017	17905	17983	17905
query5	19738	6422	6497	6422
query6	285	218	223	218
query7	4177	304	316	304
query8	445	382	397	382
query9	3139	2701	2635	2635
query10	418	316	308	308
query11	11215	10670	10698	10670
query12	132	77	70	70
query13	5594	678	696	678
query14	17697	13421	13591	13421
query15	358	233	253	233
query16	6439	289	271	271
query17	1529	1465	887	887
query18	2263	413	413	413
query19	211	156	158	156
query20	82	76	83	76
query21	187	95	94	94
query22	5218	5052	5016	5016
query23	32331	31768	31729	31729
query24	6985	6474	6527	6474
query25	519	432	421	421
query26	527	166	167	166
query27	1912	295	295	295
query28	6165	2340	2317	2317
query29	2804	2774	2642	2642
query30	245	166	176	166
query31	920	712	742	712
query32	68	65	63	63
query33	405	259	239	239
query34	847	476	475	475
query35	1163	946	936	936
query36	1299	1106	1183	1106
query37	89	61	60	60
query38	3043	2948	2926	2926
query39	1383	1311	1314	1311
query40	199	95	100	95
query41	53	44	44	44
query42	85	85	86	85
query43	842	660	654	654
query44	1125	715	722	715
query45	251	236	234	234
query46	1217	965	947	947
query47	1848	1782	1775	1775
query48	1012	701	685	685
query49	618	370	378	370
query50	867	592	605	592
query51	4833	4587	4661	4587
query52	88	91	80	80
query53	449	326	324	324
query54	2631	2450	2501	2450
query55	101	85	82	82
query56	250	224	226	224
query57	1241	1081	1188	1081
query58	214	211	196	196
query59	4119	3908	3904	3904
query60	210	211	200	200
query61	99	98	98	98
query62	830	443	494	443
query63	486	353	343	343
query64	2609	1547	1492	1492
query65	3620	3550	3562	3550
query66	800	373	381	373
query67	17233	15428	15681	15428
query68	11777	657	646	646
query69	603	344	349	344
query70	2087	1387	1426	1387
query71	431	313	313	313
query72	6594	3518	3507	3507
query73	1610	319	312	312
query74	6285	5864	5769	5769
query75	5580	3714	3727	3714
query76	6919	1120	1149	1120
query77	1186	249	257	249
query78	12581	12559	11909	11909
query79	10443	636	634	634
query80	650	410	402	402
query81	465	233	237	233
query82	236	100	96	96
query83	190	135	130	130
query84	252	74	73	73
query85	884	325	323	323
query86	346	282	286	282
query87	3329	3066	3056	3056
query88	5139	2323	2339	2323
query89	455	283	302	283
query90	2661	213	220	213
query91	173	138	142	138
query92	59	54	55	54
query93	6024	574	563	563
query94	1269	218	217	217
query95	1101	1058	1044	1044
query96	636	327	327	327
query97	6572	6346	6353	6346
query98	186	175	171	171
query99	2937	891	892	891
Total cold run time: 321822 ms
Total hot run time: 203469 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d78c3ef1aa9789e28a5383df3d6c877b1bff75e, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.25	0.04	0.04
query4	1.79	0.07	0.06
query5	0.53	0.53	0.52
query6	1.24	0.62	0.63
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.53	0.50	0.48
query10	0.53	0.55	0.54
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.62	0.61	0.60
query14	0.78	0.78	0.78
query15	0.77	0.76	0.76
query16	0.37	0.36	0.37
query17	0.98	1.02	1.02
query18	0.23	0.26	0.24
query19	1.89	1.76	1.80
query20	0.01	0.00	0.01
query21	15.47	0.56	0.58
query22	2.13	2.94	1.48
query23	16.64	1.11	0.86
query24	4.85	1.11	1.43
query25	0.38	0.10	0.05
query26	0.62	0.16	0.15
query27	0.05	0.05	0.03
query28	7.80	0.71	0.71
query29	12.62	2.31	2.25
query30	0.60	0.53	0.53
query31	2.80	0.40	0.38
query32	3.35	0.49	0.50
query33	3.09	3.08	3.06
query34	15.24	4.86	4.81
query35	4.89	4.86	4.86
query36	1.06	1.02	1.02
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.13	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 102.87 s
Total hot run time: 30.61 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7d78c3ef1aa9789e28a5383df3d6c877b1bff75e with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.3 seconds inserted 10000000 Rows, about 448K ops/s

@morningman morningman merged commit 4c4f0be into apache:branch-2.0 Jul 1, 2024
24 of 25 checks passed
@zy-kkk zy-kkk mentioned this pull request Jul 15, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zy-kkk
[branch-2.0][improvement](sqlserver catalog) Can configure whether to…
e83c6ac 
… use encryption when connecting to SQL Server using the catalog (apache#36940)

pick apache#36659
pick apache#37015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zy-kkk @doris-robot @morningman