storage: sign URL with gcloud application default credentials

[wkalt]

I would like to be able to create signed links using my application default credentials in a local development context, similar to how the library allows me to read and write to buckets without configuring separate authentication.

I believe some related work in #4604 may have addressed this for VMs running in GCE, but it does not seem to work when running locally under my gcloud login.

I would expect this code to work:

func main() {
	ctx := context.Background()
	bucket := "mybucket"
	filename := "myobject"
	method := "PUT"
	expires := time.Now().Add(time.Second * 60).Unix()
	client, err := storage.NewClient(ctx)
	if err != nil {
		log.Fatal(err)
	}
	url, err := client.Bucket(bucket).SignedURL(filename, &storage.SignedURLOptions{
		Method:  method,
		Expires: time.Unix(expires, 0),
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("URL:", url)
}

however it returns this error:

storage: unable to detect default GoogleAccessID: storage: empty client email in credentials

Is there any limitation that prevents this from working (or bug in my code)? I have owner on the project and write access to the bucket through the lib, so it seems like it should be possible for me to sign links.

My goal is to avoid developers working on this code needing to manage GOOGLE_APPLICATION_CREDENTIALS or a service account key, separate from their gcloud login.

[codyoss]

IIRC in this case you need to set the GoogleAccessID field. This can only be detected if you are using an SA or compute for auth. Then you will be using an impersonation flow which requires some IAM permissions and proper iamcredentails API enabled. If those items are done I believe this should work, but I am going off of memory 😄

[tritone]

@wkalt just to confirm, are you able to make requests using the client itself using the auth you provided? (e.g. successfully call client.Bucket().Attrs())

@codyoss if this is accurate, then I think we should update the docs for the method accordingly at least.

[codyoss]

@tritone Took a quick look at the code and I believe what I said stands. This is all alluded to in the third paragraph of https://pkg.go.dev/cloud.google.com/go/storage#BucketHandle.SignedURL, but maybe we could be even more explicit here.

[willie]

Same problem here. I've long been developing with a PrivateKey, but I'm trying super hard to move my local and deployments (Cloud Run) to default credentials for simplicity and security.

As I understand it, I'll have to pass in my email address for the GoogleAccessID field for local testing since default a. I don't understand the third paragraph's instructions, as when I click the link, that panel only appears to work with service accounts. I'm sorry for being dense here.

[tritone]

Update: we're working on supporting this without requiring passing in GoogleAccessID separately; @BrennaEpp should have a PR up soon.

[BrennaEpp]

Hi all, the PR is now merged and the changes should be available in the next release.

@willie a service account is needed to sign a URL. When you log in with gcloud, GCS can use those user credentials to perform bucket and object operations; however, signing a URL requires more than just user credentials. By default gcloud does not have a service account attached (unlike GCE, for example, which has a service account attached to the environment), making it necessary, as you mention, to pass in a service account email for the GoogleAccessID field (or, with the new changes, to login with impersonation -- see below).

@wkalt The new changes add support for detecting the service account/GoogleAccessID when logged in to gcloud application-default with the --impersonate-service-account flag set (since we need a service account to sign the url). This allows you to not have to manage a private key locally nor change your code to pass in GoogleAccessID. Just make sure, as always, that you follow the best practices for working with service accounts.

Closing this issue now; feel free to re-open if you have more questions.

[willie]

I'm not sure how to make this work for local development and continue to be able to use my default credentials for Berglas secrets. If I use the --impersonate-service-account flag, it ends up causing an error for Berglas. This might not be something you all can resolve. For now, I'm just going to use the service account key instead.

[BrennaEpp]

@willie Thank you for checking the functionality. That sounds like an issue with Berglas (possibly a simple case of lack of support for impersonation), I would suggest opening an issue there. Feel free to leave more details here as well, but as you say, it may not be something we can resolve on the Cloud Storage side.