Requester Pays on NIO

[Horneth]

There is an option on BlobSourceOption and BlobWriteOption to specify the billing project to use when accessing a requester pays bucket, but there doesn't seem to be a way to wire that through NIO.
Am I missing it ? And if not is this something that could be considered ?

[Horneth]

A more specific suggestion would be to add the relevant options for userProject: BlobSourceOption, BlobGetOption, BlobWriteOption
to all calls to the Storage object such that the billing project is always explicitly set. Would there be any objection to that ? I'm happy to make a PR.
I'm thinking here, and where necessary in CloudStorageFileSystemProvider.

This is problematic for us because it prevents using of any requester pays enabled buckets through NIO.

@jean-philippe-martin any thoughts ? :)

[jean-philippe-martin]

That's a good idea, I would welcome a PR yes. Make sure to mention me in it so I can find it easily!

[droazen]

@Horneth Do you have a PR for this feature ready to go, or could you easily make one?

[jean-philippe-martin]

Code is now in, the feature should be available. Let me know how it goes, and mention me in a new bug if you run into any difficulty!

[jean-philippe-martin]

@hzyi-google, would you mind please closing this issue? I am not able to do it myself.

[yihanzhen]

Sure. Thanks for contributing!

[jean-philippe-martin]

Thank you, @hzyi-google !

[yihanzhen]

@jean-philippe-martin There are integration tests failing. Can you double check?

[jean-philippe-martin]

Yes I can look into it @hzyi-google . Do you have something a bit more specific? Do you mean the ITStorageTest test suite?

[yihanzhen]

From my side, testCantCreateWithoutUserProject, testCantReadWithoutUserProject, testCantCopyWithoutUserProject and testFileExistsRequesterPaysNoUserProject are failing (the last one is an error).

[jean-philippe-martin]

Odd, I synced to head and those all work for me. These tests require GOOGLE_APPLICATION_CREDENTIALS to be set, but if that's it then I'd expect the failure to happen early, in the beforeClass method.

I wonder if it's something special about the test machine configuration, or the chosen project.
Requester-pays buckets require either the billing project to be set, or the project to have the resourcemanager.projects.createBillingAssignment permission. Could it be that this permission is enabled? Then the accesses would pass, causing the test to fail.

Can you share the error for the last test (either here or via email if you prefer)? Perhaps that'll give me a hint about how to reproduce the issue.

[yihanzhen]

@jean-philippe-martin Yes you are right. Our testing project do have the permission to the method in resourcemanager api. And the test with errors is for the same reason. I'll create a separate github issue for this and ignore the tests for now. Thanks for the clarification :)

[jean-philippe-martin]

@hzyi-google great, I'm glad the mystery is solved!

Hopefully there is some way you can use a project without these permissions, so you can restore the tests on your side. Or split the integration tests into two parts, one with that permission and one without.