Support read-only cache

[MarcoPolo]

In draft until I try this branch out somewhere

[marten-seemann]

What's the use case for this?

[MarcoPolo]

Other implementations can benefit without having write access to the cache bucket (which would be a security concern).

[marten-seemann]

Other implementations

Who specifically would use this? Is this for @thomaseizinger who's clearing his caches (too) frequently, or are there any other users?

[MarcoPolo]

zig-libp2p is using this. This will also really help nim-libp2p folks since a recent build of there's took ~30min: https://github.com/status-im/nim-libp2p/actions/runs/4385644603/jobs/7678638102.

At some point without caching the runners would run out of space. Building all these things takes a bit of a space.

[MarcoPolo]

cc @Menduist . This will speed up these jobs by about 20min. Ping me on slack and I'll share the read only keys :)

[marten-seemann]

zig-libp2p is using this. This will also really help nim-libp2p folks since a recent build of there's took ~30min: https://github.com/status-im/nim-libp2p/actions/runs/4385644603/jobs/7678638102.

At some point without caching the runners would run out of space. Building all these things takes a bit of a space.

Thanks! Sounds good to me, let's ship it!

[thomaseizinger]

Thanks @MarcoPolo , this has been on my list for a while and I didn't get to it.

I did envision it slightly differently though. At the moment, CI runs from forks of our repos don't have access to the secrets and thus don't have access to the caches, right?

I figured instead of opting into a read-only cache, we should make that the default and configure the read-only keys as regular variables in our CI to allow forks access to it.

Writing to the cache should then be conditional on the actual secret being present. What do you think?

[MarcoPolo]

I figured instead of opting into a read-only cache, we should make that the default and configure the read-only keys as regular variables in our CI to allow forks access to it.

True. I had some issue with this, but I'd like this to work as well. I believe the bucket is public to list, but not to get. I'll try to get this working.

[MarcoPolo]

Blocked on:

• Make build cache bucket public #132 (comment)

[MarcoPolo]

Ugh, unfortunately you cannot simply omit the credentials. The AWS sdk wants the credentials to make signed requests to S3 (see the note here). I was able to create a new IAM user with the following permissions on my personal account and have it work:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "arn:aws:s3:::libp2p-by-tf-aws-bootstrap/*"
        }
    ]
}

Not as nice as seamlessly working with 0 setup. But at least it can be configured by individuals.

[thomaseizinger]

That is kind of annoying because requests via HTTP work without any setup?

[MarcoPolo]

Yup!

[galargh]

This will allow using anonymous creds for public buckets: moby/buildkit#3692

[thomaseizinger]

Ugh, unfortunately you cannot simply omit the credentials. The AWS sdk wants the credentials to make signed requests to S3 (see the note here). I was able to create a new IAM user with the following permissions on my personal account and have it work:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "arn:aws:s3:::libp2p-by-tf-aws-bootstrap/*"
        }
    ]
}

Not as nice as seamlessly working with 0 setup. But at least it can be configured by individuals.

Can we create a set of credentials that we default to?

To properly make this work for forks, we need it to be the same workflow, i.e. we can't specify a flag on the action whether we want to read or read-write.

Instead, I'd suggest that we default to a read-only user in the dockerBuildWrapper.sh script. That would also allow us to use this cache locally. In case a user is specified via the env variables, we override it and expect the credentials to have read-write permissions.

Thoughts?

[galargh]

Personally, I'd wait for moby/buildkit#3692 to be merged and then I'd start using the buildkit version from master. It's already approved so hopefully it's not long until it's merged.

[thomaseizinger]

Personally, I'd wait for moby/buildkit#3692 to be merged and then I'd start using the buildkit version from master. It's already approved so hopefully it's not long until it's merged.

Sounds good.

[MarcoPolo]

With #165 this is done