Backup to Google Cloud Storage

[filiprafaj]

Hello, I am trying and failing to store backups to Google Cloud Storage.

I have set up Workload Identity to give the k8s service account the permissions to access the bucket.

I have also tried defining the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables.

Still I get
Error: failed to take a full dump: failed to put dump.tar: operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

Is there anybody who made backups to GCS working?

Thank you!

[ymmt2005]

Backup to GCS is not supported now.
It can be added by implementing Bucket interface for GCS in this package.
https://github.com/cybozu-go/moco/tree/main/pkg/bucket

We welcome a pull request for adding GCS support.

[d-kuro]

@filiprafaj
GCS supports the S3 compatibility API. (Sorry, I have not verified this.)
Could you please refer to the documentation and try again?

refs:

• https://cloud.google.com/storage/docs/interoperability#xml_api
• https://vamsiramakrishnan.medium.com/a-study-on-using-google-cloud-storage-with-the-s3-compatibility-api-324d31b8dfeb

If it still doesn't work, it would be helpful if you could report it again, including the definition of BackupPolicy 🙏
https://cybozu-go.github.io/moco/usage.html#backuppolicy

[filiprafaj]

Hi @d-kuro , I have tried now with interoperability credentials and I am getting:

Error: failed to take a full dump: failed to put dump.tar: operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method. failed to take a full dump: failed to put dump.tar: operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

the backup file looks like this:

apiVersion: moco.cybozu.com/v1beta2
kind: BackupPolicy
metadata:
  namespace: default
  name: daily
spec:
  schedule: "@daily"
  jobConfig:
    serviceAccountName: moco-test-mysqlcluster
    env:
    - name: AWS_ACCESS_KEY_ID
      value: ***
    - name: AWS_SECRET_ACCESS_KEY
      value: ***
    bucketConfig:
      bucketName: ***
      endpointURL: https://storage.googleapis.com
    workVolume:
      emptyDir: {}
    memory: 1Gi
    maxMemory: 1Gi
    threads: 1

[Prakharkarsh1]

Hii @filiprafaj i want to contribute to this issue

[ymmt2005]

@Prakharkarsh1
Hi,
Thank you for your intention to contribute to this project.
We will review your pull request when it's ready.

[yamatcha]

@Prakharkarsh1

MOCO uses aws-sdk-go-v2 to connect to s3-compatible storage.
However, aws-sdk-go-v2 is not compatible with third-party platforms and therefore cannot connect GCS.
aws/aws-sdk-go-v2#1816
So it would be better to implement for gcs bucket in moco/pkg/bucket/gcs.

[ymmt2005]

We use minio to test S3 bucket implementation.

Likewise, we may use these tools to test GCS bucket implementation.

• https://github.com/oittaa/gcp-storage-emulator
• https://github.com/fsouza/fake-gcs-server

[ymmt2005]

@Prakharkarsh1
Hello,
Do you still want to contribute to this feature?

[masa213f]

Released https://github.com/cybozu-go/moco/releases/tag/v0.16.1