Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Presto support COS select #4519

Closed
Coderlxl opened this issue Jul 21, 2020 · 2 comments
Closed

Presto support COS select #4519

Coderlxl opened this issue Jul 21, 2020 · 2 comments

Comments

@Coderlxl
Copy link

Coderlxl commented Jul 21, 2020

We used Presto to test COS Select feature on csv files, but it‘s failed.

We checked Presto source code and COS documentation, found that it was caused by the following reasons:

  1. COS Select feature only support virtual-hosted style URL. But Presto only support path-style URL. The hard code is in PrestoS3ClientFactory class.
    image

  2. Cos Select feature don't support csv comments = "\uFDD0", but it's hard code in S3SelectCsvRecordReader class.
    image

So, I think these parameters should be configurable in Presto. My patch fixed this problem.
0001-support-cos-select.patch.txt

Test fixed result:
(1) Fix Before, Presto's hive.properties is:
hive.s3.aws-access-key=AK
hive.s3.aws-secret-key=SK
hive.s3.endpoint=https://cos.ap-beijing.myqcloud.com
hive.s3select-pushdown.enabled=true
#hive.s3.path-style-access=false
#hive.s3.csv-comments=#

failed result
image

(2) Fix After, Presto's hive.properties is:
hive.s3.aws-access-key=AK
hive.s3.aws-secret-key=SK
hive.s3.endpoint=https://cos.ap-beijing.myqcloud.com
hive.s3select-pushdown.enabled=true
hive.s3.path-style-access=false
hive.s3.csv-comments=#

succeed result
image

COS Reference Documentation https://intl.cloud.tencent.com/document/product/436/32472

@electrum
Copy link
Member

Thanks for filing this issue and including a patch. Can you sign the CLA and submit this patch as a pull request?

@hashhar
Copy link
Member

hashhar commented Feb 22, 2024

s3 select was removed in #18241 and I don't think we'll be adding COS Select because of the direction filesystem code is moving in.

@hashhar hashhar closed this as not planned Won't fix, can't repro, duplicate, stale Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

3 participants