Skip to content

Commit

Permalink
feat(object): basic CRUD support for buckets (#3588)
Browse files Browse the repository at this point in the history
Co-authored-by: scaleway-bot <github@scaleway.com>
Co-authored-by: lmasson <lmasson@scaleway.com>
Co-authored-by: Laure-di <62625835+Laure-di@users.noreply.github.com>
Co-authored-by: Rémy Léone <rleone@scaleway.com>
  • Loading branch information
5 people authored Jul 1, 2024
1 parent 729c9ab commit 6790095
Show file tree
Hide file tree
Showing 39 changed files with 5,482 additions and 19 deletions.
22 changes: 22 additions & 0 deletions cmd/scw/testdata/test-all-usage-object-bucket-create-usage.golden
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
🎲🎲🎲 EXIT CODE: 0 🎲🎲🎲
🟥🟥🟥 STDERR️️ 🟥🟥🟥️
Create an Object Storage Bucket with the S3 protocol. The namespace is shared between all S3 users, so its name must be unique.

USAGE:
scw object bucket create <name ...> [arg=value ...]

ARGS:
name The unique name of the bucket
[tags.{index}] List of tags to set on the bucket
[enable-versioning=false] Whether or not objects in the bucket should have multiple versions
[acl=private] The permissions given to users (grantees) to read or write objects
[region=fr-par] Region to target. If none is passed will use default region from the config

FLAGS:
-h, --help help for create

GLOBAL FLAGS:
-c, --config string The path to the config file
-D, --debug Enable debug mode
-o, --output string Output format: json or human, see 'scw help output' for more info (default "human")
-p, --profile string The config profile to use
19 changes: 19 additions & 0 deletions cmd/scw/testdata/test-all-usage-object-bucket-delete-usage.golden
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
🎲🎲🎲 EXIT CODE: 0 🎲🎲🎲
🟥🟥🟥 STDERR️️ 🟥🟥🟥️
Delete an S3 bucket with all its content.

USAGE:
scw object bucket delete <name ...> [arg=value ...]

ARGS:
name The unique name of the bucket
[region=fr-par] Region to target. If none is passed will use default region from the config

FLAGS:
-h, --help help for delete

GLOBAL FLAGS:
-c, --config string The path to the config file
-D, --debug Enable debug mode
-o, --output string Output format: json or human, see 'scw help output' for more info (default "human")
-p, --profile string The config profile to use
20 changes: 20 additions & 0 deletions cmd/scw/testdata/test-all-usage-object-bucket-get-usage.golden
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
🎲🎲🎲 EXIT CODE: 0 🎲🎲🎲
🟥🟥🟥 STDERR️️ 🟥🟥🟥️
Get the properties of an S3 bucket like tags, endpoint, access control, versioning, size, etc.

USAGE:
scw object bucket get <name ...> [arg=value ...]

ARGS:
name The unique name of the bucket
[with-size=false] Whether to return the total size of the bucket and the number of objects. This operation can take long for large buckets.
[region=fr-par] Region to target. If none is passed will use default region from the config

FLAGS:
-h, --help help for get

GLOBAL FLAGS:
-c, --config string The path to the config file
-D, --debug Enable debug mode
-o, --output string Output format: json or human, see 'scw help output' for more info (default "human")
-p, --profile string The config profile to use
18 changes: 18 additions & 0 deletions cmd/scw/testdata/test-all-usage-object-bucket-list-usage.golden
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
🎲🎲🎲 EXIT CODE: 0 🎲🎲🎲
🟥🟥🟥 STDERR️️ 🟥🟥🟥️
List all existing S3 buckets in the specified region

USAGE:
scw object bucket list [arg=value ...]

ARGS:
[region=fr-par] Region to target. If none is passed will use default region from the config

FLAGS:
-h, --help help for list

GLOBAL FLAGS:
-c, --config string The path to the config file
-D, --debug Enable debug mode
-o, --output string Output format: json or human, see 'scw help output' for more info (default "human")
-p, --profile string The config profile to use
22 changes: 22 additions & 0 deletions cmd/scw/testdata/test-all-usage-object-bucket-update-usage.golden
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
🎲🎲🎲 EXIT CODE: 0 🎲🎲🎲
🟥🟥🟥 STDERR️️ 🟥🟥🟥️
Update an S3 bucket's properties like tags, access control and versioning.

USAGE:
scw object bucket update <name ...> [arg=value ...]

ARGS:
name The unique name of the bucket
[tags.{index}] List of new tags to set on the bucket
[enable-versioning=false] Whether or not objects in the bucket should have multiple versions
[acl=private] The permissions given to users (grantees) to read or write objects
[region=fr-par] Region to target. If none is passed will use default region from the config

FLAGS:
-h, --help help for update

GLOBAL FLAGS:
-c, --config string The path to the config file
-D, --debug Enable debug mode
-o, --output string Output format: json or human, see 'scw help output' for more info (default "human")
-p, --profile string The config profile to use
24 changes: 24 additions & 0 deletions cmd/scw/testdata/test-all-usage-object-bucket-usage.golden
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
🎲🎲🎲 EXIT CODE: 0 🎲🎲🎲
🟥🟥🟥 STDERR️️ 🟥🟥🟥️
Manage S3 buckets creation, deletion and updates to properties like tags, ACL and versioning.

USAGE:
scw object bucket <command>

AVAILABLE COMMANDS:
create Create an S3 bucket
delete Delete an S3 bucket
get Get information about an S3 bucket
list List S3 buckets
update Update an S3 bucket

FLAGS:
-h, --help help for bucket

GLOBAL FLAGS:
-c, --config string The path to the config file
-D, --debug Enable debug mode
-o, --output string Output format: json or human, see 'scw help output' for more info (default "human")
-p, --profile string The config profile to use

Use "scw object bucket [command] --help" for more information about a command.
1 change: 1 addition & 0 deletions cmd/scw/testdata/test-all-usage-object-usage.golden
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ USAGE:
scw object <command>

AVAILABLE COMMANDS:
bucket Manage S3 buckets
config Manage configuration files for popular S3 tools

FLAGS:
Expand Down
117 changes: 117 additions & 0 deletions docs/commands/object.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,128 @@
# Documentation for `scw object`
Object-storage utils

- [Manage S3 buckets](#manage-s3-buckets)
- [Create an S3 bucket](#create-an-s3-bucket)
- [Delete an S3 bucket](#delete-an-s3-bucket)
- [Get information about an S3 bucket](#get-information-about-an-s3-bucket)
- [List S3 buckets](#list-s3-buckets)
- [Update an S3 bucket](#update-an-s3-bucket)
- [Manage configuration files for popular S3 tools](#manage-configuration-files-for-popular-s3-tools)
- [Generate a S3 tool configuration file](#generate-a-s3-tool-configuration-file)
- [Install a S3 tool configuration file to its default location](#install-a-s3-tool-configuration-file-to-its-default-location)


## Manage S3 buckets

Manage S3 buckets creation, deletion and updates to properties like tags, ACL and versioning.


### Create an S3 bucket

Create an Object Storage Bucket with the S3 protocol. The namespace is shared between all S3 users, so its name must be unique.

**Usage:**

```
scw object bucket create <name ...> [arg=value ...]
```


**Args:**

| Name | | Description |
|------|---|-------------|
| name | Required | The unique name of the bucket |
| tags.{index} | | List of tags to set on the bucket |
| enable-versioning | Default: `false` | Whether or not objects in the bucket should have multiple versions |
| acl | Default: `private` | The permissions given to users (grantees) to read or write objects |
| region | Default: `fr-par` | Region to target. If none is passed will use default region from the config |



### Delete an S3 bucket

Delete an S3 bucket with all its content.

**Usage:**

```
scw object bucket delete <name ...> [arg=value ...]
```


**Args:**

| Name | | Description |
|------|---|-------------|
| name | Required | The unique name of the bucket |
| region | Default: `fr-par` | Region to target. If none is passed will use default region from the config |



### Get information about an S3 bucket

Get the properties of an S3 bucket like tags, endpoint, access control, versioning, size, etc.

**Usage:**

```
scw object bucket get <name ...> [arg=value ...]
```


**Args:**

| Name | | Description |
|------|---|-------------|
| name | Required | The unique name of the bucket |
| with-size | Default: `false` | Whether to return the total size of the bucket and the number of objects. This operation can take long for large buckets. |
| region | Default: `fr-par` | Region to target. If none is passed will use default region from the config |



### List S3 buckets

List all existing S3 buckets in the specified region

**Usage:**

```
scw object bucket list [arg=value ...]
```


**Args:**

| Name | | Description |
|------|---|-------------|
| region | Default: `fr-par` | Region to target. If none is passed will use default region from the config |



### Update an S3 bucket

Update an S3 bucket's properties like tags, access control and versioning.

**Usage:**

```
scw object bucket update <name ...> [arg=value ...]
```


**Args:**

| Name | | Description |
|------|---|-------------|
| name | Required | The unique name of the bucket |
| tags.{index} | | List of new tags to set on the bucket |
| enable-versioning | Default: `false` | Whether or not objects in the bucket should have multiple versions |
| acl | Default: `private` | The permissions given to users (grantees) to read or write objects |
| region | Default: `fr-par` | Region to target. If none is passed will use default region from the config |



## Manage configuration files for popular S3 tools

Configuration generation for S3 tools.
Expand Down
17 changes: 11 additions & 6 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ toolchain go1.22.2

require (
github.com/alecthomas/assert v1.0.0
github.com/aws/aws-sdk-go-v2 v1.26.1
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.1
github.com/buildpacks/pack v0.34.2
github.com/c-bata/go-prompt v0.2.6
github.com/charmbracelet/bubbletea v0.26.6
Expand Down Expand Up @@ -58,21 +60,24 @@ require (
github.com/alecthomas/colour v0.1.0 // indirect
github.com/alecthomas/repr v0.2.0 // indirect
github.com/apex/log v1.9.0 // indirect
github.com/aws/aws-sdk-go-v2 v1.26.0 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
github.com/aws/aws-sdk-go-v2/config v1.27.7 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.7 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.5 // indirect
github.com/aws/aws-sdk-go-v2/service/ecr v1.24.5 // indirect
github.com/aws/aws-sdk-go-v2/service/ecrpublic v1.21.5 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.5 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.20.2 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.2 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.28.4 // indirect
github.com/aws/smithy-go v1.20.1 // indirect
github.com/aws/smithy-go v1.20.2 // indirect
github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20231213181459-b0fcec718dc6 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/buildpacks/imgutil v0.0.0-20240605145725-186f89b2d168 // indirect
Expand Down
34 changes: 22 additions & 12 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -66,36 +66,46 @@ github.com/aphistic/sweet v0.2.0/go.mod h1:fWDlIh/isSE9n6EPsRmC0det+whmX6dJid3st
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
github.com/aws/aws-sdk-go v1.20.6/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/aws/aws-sdk-go-v2 v1.26.0 h1:/Ce4OCiM3EkpW7Y+xUnfAFpchU78K7/Ug01sZni9PgA=
github.com/aws/aws-sdk-go-v2 v1.26.0/go.mod h1:35hUlJVYd+M++iLI3ALmVwMOyRYMmRqUXpTtRGW+K9I=
github.com/aws/aws-sdk-go-v2 v1.26.1 h1:5554eUqIYVWpU0YmeeYZ0wU64H2VLBs8TlhRB2L+EkA=
github.com/aws/aws-sdk-go-v2 v1.26.1/go.mod h1:ffIFB97e2yNsv4aTSGkqtHnppsIJzw7G7BReUZ3jCXM=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 h1:x6xsQXGSmW6frevwDA+vi/wqhp1ct18mVXYN08/93to=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2/go.mod h1:lPprDr1e6cJdyYeGXnRaJoP4Md+cDBvi2eOj00BlGmg=
github.com/aws/aws-sdk-go-v2/config v1.27.7 h1:JSfb5nOQF01iOgxFI5OIKWwDiEXWTyTgg1Mm1mHi0A4=
github.com/aws/aws-sdk-go-v2/config v1.27.7/go.mod h1:PH0/cNpoMO+B04qET699o5W92Ca79fVtbUnvMIZro4I=
github.com/aws/aws-sdk-go-v2/credentials v1.17.7 h1:WJd+ubWKoBeRh7A5iNMnxEOs982SyVKOJD+K8HIezu4=
github.com/aws/aws-sdk-go-v2/credentials v1.17.7/go.mod h1:UQi7LMR0Vhvs+44w5ec8Q+VS+cd10cjwgHwiVkE0YGU=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.3 h1:p+y7FvkK2dxS+FEwRIDHDe//ZX+jDhP8HHE50ppj4iI=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.3/go.mod h1:/fYB+FZbDlwlAiynK9KDXlzZl3ANI9JkD0Uhz5FjNT4=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.3 h1:ifbIbHZyGl1alsAhPIYsHOg5MuApgqOvVeI8wIugXfs=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.3/go.mod h1:oQZXg3c6SNeY6OZrDY+xHcF4VGIEoNotX2B4PrDeoJI=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.3 h1:Qvodo9gHG9F3E8SfYOspPeBt0bjSbsevK8WhRAUHcoY=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.3/go.mod h1:vCKrdLXtybdf/uQd/YfVR2r5pcbNuEYKzMQpcxmeSJw=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.5 h1:aw39xVGeRWlWx9EzGVnhOR4yOjQDHPQ6o6NmBlscyQg=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.5/go.mod h1:FSaRudD0dXiMPK2UjknVwwTYyZMRsHv3TtkabsZih5I=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5 h1:PG1F3OD1szkuQPzDw3CIQsRIrtTlUC3lP84taWzHlq0=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5/go.mod h1:jU1li6RFryMz+so64PpKtudI+QzbKoIEivqdf6LNpOc=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 h1:hT8rVHwugYE2lEfdFE0QWVo81lF7jMrYJVDWI+f+VxU=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0/go.mod h1:8tu/lYfQfFe6IGnaOdrpVgEL2IrrDOf6/m9RQum4NkY=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.5 h1:81KE7vaZzrl7yHBYHVEzYB8sypz11NMOZ40YlWvPxsU=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.5/go.mod h1:LIt2rg7Mcgn09Ygbdh/RdIm0rQ+3BNkbP1gyVMFtRK0=
github.com/aws/aws-sdk-go-v2/service/ecr v1.24.5 h1:wLPDAUFT50NEXGXpywRU3AA74pg35RJjWol/68ruvQQ=
github.com/aws/aws-sdk-go-v2/service/ecr v1.24.5/go.mod h1:AOHmGMoPtSY9Zm2zBuwUJQBisIvYAZeA1n7b6f4e880=
github.com/aws/aws-sdk-go-v2/service/ecrpublic v1.21.5 h1:PQp21GBlGNaQ+AVJAB8w2KTmLx0DkFS2fDET2Iy3+f0=
github.com/aws/aws-sdk-go-v2/service/ecrpublic v1.21.5/go.mod h1:WMntdAol8KgeYsa5sDZPsRTXs4jVZIMYu0eQVVIQxnc=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1 h1:EyBZibRTVAs6ECHZOw5/wlylS9OcTzwyjeQMudmREjE=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1/go.mod h1:JKpmtYhhPs7D97NL/ltqz7yCkERFW5dOlHyVl66ZYF8=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.5 h1:K/NXvIftOlX+oGgWGIa3jDyYLDNsdVhsjHmsBH2GLAQ=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.5/go.mod h1:cl9HGLV66EnCmMNzq4sYOti+/xo8w34CsgzVtm2GgsY=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2 h1:Ji0DY1xUsUr3I8cHps0G+XM3WWU16lP6yG8qu1GAZAs=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2/go.mod h1:5CsjAbs3NlGQyZNFACh+zztPDI7fU6eW9QsxjfnuBKg=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.7 h1:ZMeFZ5yk+Ek+jNr1+uwCd2tG89t6oTS5yVWpa6yy2es=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.7/go.mod h1:mxV05U+4JiHqIpGqqYXOHLPKUC6bDXC44bsUhNjOEwY=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.7 h1:ogRAwT1/gxJBcSWDMZlgyFUM962F51A5CRhDLbxLdmo=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.7/go.mod h1:YCsIZhXfRPLFFCl5xxY+1T9RKzOKjCut+28JSX2DnAk=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.5 h1:f9RyWNtS8oH7cZlbn+/JNPpjUk5+5fLd5lM9M0i49Ys=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.5/go.mod h1:h5CoMZV2VF297/VLhRhO1WF+XYWOzXo+4HsObA4HjBQ=
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.1 h1:6cnno47Me9bRykw9AEv9zkXE+5or7jz8TsskTTccbgc=
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.1/go.mod h1:qmdkIIAC+GCLASF7R2whgNrJADz0QZPX+Seiw/i4S3o=
github.com/aws/aws-sdk-go-v2/service/sso v1.20.2 h1:XOPfar83RIRPEzfihnp+U6udOveKZJvPQ76SKWrLRHc=
github.com/aws/aws-sdk-go-v2/service/sso v1.20.2/go.mod h1:Vv9Xyk1KMHXrR3vNQe8W5LMFdTjSeWk0gBZBzvf3Qa0=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.2 h1:pi0Skl6mNl2w8qWZXcdOyg197Zsf4G97U7Sso9JXGZE=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.2/go.mod h1:JYzLoEVeLXk+L4tn1+rrkfhkxl6mLDEVaDSvGq9og90=
github.com/aws/aws-sdk-go-v2/service/sts v1.28.4 h1:Ppup1nVNAOWbBOrcoOxaxPeEnSFB2RnnQdguhXpmeQk=
github.com/aws/aws-sdk-go-v2/service/sts v1.28.4/go.mod h1:+K1rNPVyGxkRuv9NNiaZ4YhBFuyw2MMA9SlIJ1Zlpz8=
github.com/aws/smithy-go v1.20.1 h1:4SZlSlMr36UEqC7XOyRVb27XMeZubNcBNN+9IgEPIQw=
github.com/aws/smithy-go v1.20.1/go.mod h1:krry+ya/rV9RDcV/Q16kpu6ypI4K2czasz0NC3qS14E=
github.com/aws/smithy-go v1.20.2 h1:tbp628ireGtzcHDDmLT/6ADHidqnwgF57XOXZe6tp4Q=
github.com/aws/smithy-go v1.20.2/go.mod h1:krry+ya/rV9RDcV/Q16kpu6ypI4K2czasz0NC3qS14E=
github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20231213181459-b0fcec718dc6 h1:PlJRmqKlSlEUlwem1c3zdPaEMtJc/ktnV7naD5Qvsx4=
github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20231213181459-b0fcec718dc6/go.mod h1:08sPJIlDHu4HwQ1xScPgsBWezvM6U10ghGKBJu0mowA=
github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod h1:q/89r3U2H7sSsE2t6Kca0lfwTK8JdoNGS/yzM/4iH5I=
Expand Down
Loading

0 comments on commit 6790095

Please sign in to comment.