We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
We used Presto to test COS Select feature on csv files, but it‘s failed.
We checked Presto source code and COS documentation, found that it was caused by the following reasons:
COS Select feature only support virtual-hosted style URL. But Presto only support path-style URL. The hard code is in PrestoS3ClientFactory class.
Cos Select feature don't support csv comments = "\uFDD0", but it's hard code in S3SelectCsvRecordReader class.
So, I think these parameters should be configurable in Presto. My patch fixed this problem. 0001-support-cos-select.patch.txt
Test fixed result: (1) Fix Before, Presto's hive.properties is: hive.s3.aws-access-key=AK hive.s3.aws-secret-key=SK hive.s3.endpoint=https://cos.ap-beijing.myqcloud.com hive.s3select-pushdown.enabled=true #hive.s3.path-style-access=false #hive.s3.csv-comments=#
failed result
(2) Fix After, Presto's hive.properties is: hive.s3.aws-access-key=AK hive.s3.aws-secret-key=SK hive.s3.endpoint=https://cos.ap-beijing.myqcloud.com hive.s3select-pushdown.enabled=true hive.s3.path-style-access=false hive.s3.csv-comments=#
succeed result
COS Reference Documentation https://intl.cloud.tencent.com/document/product/436/32472
The text was updated successfully, but these errors were encountered:
Thanks for filing this issue and including a patch. Can you sign the CLA and submit this patch as a pull request?
Sorry, something went wrong.
s3 select was removed in #18241 and I don't think we'll be adding COS Select because of the direction filesystem code is moving in.
Successfully merging a pull request may close this issue.
We used Presto to test COS Select feature on csv files, but it‘s failed.
We checked Presto source code and COS documentation, found that it was caused by the following reasons:
COS Select feature only support virtual-hosted style URL. But Presto only support path-style URL. The hard code is in PrestoS3ClientFactory class.
![image](https://user-images.githubusercontent.com/7564040/88052332-b0c85380-cb8c-11ea-9627-a535f9e992db.png)
Cos Select feature don't support csv comments = "\uFDD0", but it's hard code in S3SelectCsvRecordReader class.
![image](https://user-images.githubusercontent.com/7564040/88055419-8e850480-cb91-11ea-9780-c7a9f37cb5e9.png)
So, I think these parameters should be configurable in Presto. My patch fixed this problem.
0001-support-cos-select.patch.txt
Test fixed result:
(1) Fix Before, Presto's hive.properties is:
hive.s3.aws-access-key=AK
hive.s3.aws-secret-key=SK
hive.s3.endpoint=https://cos.ap-beijing.myqcloud.com
hive.s3select-pushdown.enabled=true
#hive.s3.path-style-access=false
#hive.s3.csv-comments=#
failed result
![image](https://user-images.githubusercontent.com/7564040/88057014-0a804c00-cb94-11ea-903a-71a805020080.png)
(2) Fix After, Presto's hive.properties is:
hive.s3.aws-access-key=AK
hive.s3.aws-secret-key=SK
hive.s3.endpoint=https://cos.ap-beijing.myqcloud.com
hive.s3select-pushdown.enabled=true
hive.s3.path-style-access=false
hive.s3.csv-comments=#
succeed result
![image](https://user-images.githubusercontent.com/7564040/88057155-3d2a4480-cb94-11ea-9618-72396adde2ff.png)
COS Reference Documentation https://intl.cloud.tencent.com/document/product/436/32472
The text was updated successfully, but these errors were encountered: