SignatureDoesNotMatch error when used with GCS #1816
Comments
|
Hi @boraberke, Thanks for opening the issue. The SDKs don't guarantee compatibility with 3rd party platforms like in your case. I wish I could help. |
|
Hi @RanVaknin, I found the issue of the problem. When I remove I used Do you know the reason why |
|
Hi @boraberke , Im not entirely sure. The V2 of the SDK was released to conform to requirements from the AWS service teams. I'm happy you were able to figure it out and make it work! Hope I could be more help in the future. |
|
Hi @boraberke, could you please share how you excluded the |
Unfortunately I couldn't find a way to exclude accept-encoding without changing the source code of the sdk. |
|
Hi @boraberke ,@szabolcsgelencser, and anyone else who is also experiencing this: I don't have experience in GCS, and in order to take a deeper look I need some detailed repro steps. Thanks, |
|
This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled. |
|
@RanVaknin, we experience same issue with our code to test GCP bucket Steps to reproduce
Check credentials and permissions# Version
aws --version
aws-cli/2.9.9 Python/3.9.11
# Authenticate
export AWS_ACCESS_KEY_ID="<AWS_ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<AWS_SECRET_ACCESS_KEY>"
# List
aws s3 --endpoint-url https://storage.googleapis.com ls bucket
2023-03-13 13:32:47 3 check.txt
# Head bucket
aws s3api \
--endpoint-url https://storage.googleapis.com \
head-bucket \
--bucket bucket
# Get Region
aws s3api \
--endpoint-url https://storage.googleapis.com \
get-bucket-location \
--bucket bucket
{
"LocationConstraint": "ASIA-SOUTHEAST1"
} |
I found a little hack way to achieve it. the idea is to use |
|
More safe way, https://stackoverflow.com/a/74382598/1204665, but with more code and could be less performance |
…ackoverflow.com/a/74382598/1204665, aws/aws-sdk-go-v2#1816, GCS over S3 have no 5Gb restriction, TestIntegrationGCS pass again, TestIntegrationEmbedded need to fix
|
I'm not sure whether it will be useful to others or if it is the best way to handle it, but based on the above stackoverflow question, here is my Usage pseudo-code: client := s3.NewFromConfig(cfg, func(o *s3.Options) {
// Google Cloud Storage alters the Accept-Encoding header, which breaks the v2 request signature
// (https://github.com/aws/aws-sdk-go-v2/issues/1816)
if strings.Contains(endpoint, "storage.googleapis.com") {
ignoreSigningHeaders(o, []string{"Accept-Encoding"})
}
})Middleware: package yourpackage
import (
"context"
"fmt"
v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/smithy-go/middleware"
smithyhttp "github.com/aws/smithy-go/transport/http"
)
// ignoreSigningHeaders excludes the listed headers
// from the request signature because some providers may alter them.
//
// See https://github.com/aws/aws-sdk-go-v2/issues/1816.
func ignoreSigningHeaders(o *s3.Options, headers []string) {
o.APIOptions = append(o.APIOptions, func(stack *middleware.Stack) error {
if err := stack.Finalize.Insert(ignoreHeaders(headers), "Signing", middleware.Before); err != nil {
return err
}
if err := stack.Finalize.Insert(restoreIgnored(), "Signing", middleware.After); err != nil {
return err
}
return nil
})
}
type ignoredHeadersKey struct{}
func ignoreHeaders(headers []string) middleware.FinalizeMiddleware {
return middleware.FinalizeMiddlewareFunc(
"IgnoreHeaders",
func(ctx context.Context, in middleware.FinalizeInput, next middleware.FinalizeHandler) (out middleware.FinalizeOutput, metadata middleware.Metadata, err error) {
req, ok := in.Request.(*smithyhttp.Request)
if !ok {
return out, metadata, &v4.SigningError{Err: fmt.Errorf("(ignoreHeaders) unexpected request middleware type %T", in.Request)}
}
ignored := make(map[string]string, len(headers))
for _, h := range headers {
ignored[h] = req.Header.Get(h)
req.Header.Del(h)
}
ctx = middleware.WithStackValue(ctx, ignoredHeadersKey{}, ignored)
return next.HandleFinalize(ctx, in)
},
)
}
func restoreIgnored() middleware.FinalizeMiddleware {
return middleware.FinalizeMiddlewareFunc(
"RestoreIgnored",
func(ctx context.Context, in middleware.FinalizeInput, next middleware.FinalizeHandler) (out middleware.FinalizeOutput, metadata middleware.Metadata, err error) {
req, ok := in.Request.(*smithyhttp.Request)
if !ok {
return out, metadata, &v4.SigningError{Err: fmt.Errorf("(restoreIgnored) unexpected request middleware type %T", in.Request)}
}
ignored, _ := middleware.GetStackValue(ctx, ignoredHeadersKey{}).(map[string]string)
for k, v := range ignored {
req.Header.Set(k, v)
}
return next.HandleFinalize(ctx, in)
},
)
} |
Describe the bug
I'm facing
SignatureDoesNotMatcherror when I use endpoint url as google cloud storage https://storage.googleapis.com.I didn't have this error when using
aws-sdk-go.Expected Behavior
To be able to send requests successfully just like in
aws-sdk-go.Current Behavior
I used a simple code where I try to
ListBucketswith default settings. Following is the debug output and error:Similarly, when I try to use
aws-sdk-go, it is successful:Reproduction Steps
Below there are two gists, identically doing the same thing using
aws-sdk-goandaws-sdk-go-v2respectively.using aws-sdk-go
using aws-sdk-go-v2
Possible Solution
Some additional headers such as
amz-sdk-requestandamz-sdk-invocation-idare added toSignedHeadersin v2. Issue might be related to that, but I don't have a possible solution in mind.Additional Information/Context
GCS gives an example usage with
aws-sdk-gohere but there is no example withaws-sdk-go-v2.AWS Go SDK V2 Module Versions Used
Compiler and Version used
go version go1.18.3 darwin/amd64
Operating System and version
macos monterey 12.4
The text was updated successfully, but these errors were encountered: