From b166c44f65ed8de2e167989ae25e148186758c55 Mon Sep 17 00:00:00 2001 From: ash Date: Thu, 9 Jan 2025 15:28:26 +0000 Subject: [PATCH 1/5] Consolidating `w3up` and `upload-service` --- rfc/consolidating-w3up-and-upload-service.md | 72 ++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 rfc/consolidating-w3up-and-upload-service.md diff --git a/rfc/consolidating-w3up-and-upload-service.md b/rfc/consolidating-w3up-and-upload-service.md new file mode 100644 index 0000000..47ee0b5 --- /dev/null +++ b/rfc/consolidating-w3up-and-upload-service.md @@ -0,0 +1,72 @@ +# RFC: Consolidating `w3up` + `upload-service` + +## Introduction + +For the event held in Bangkok, I forked `w3up` and `w3infra` and deployed a new service, and modules, with some changes to allow the w3up service to store data on decentralized storage nodes. I named the new repos `upload-service` and `upload-service-infra`. + +We now need to plan a route forward to consolitate these services without breaking compatibility. This RFC is a proposal of that plan. + +_Not in scope: this is not a proposal for monorepoing all the things. This is a proposal for how to remove the repo duplication in the existing arrangement of things._ + +### What is the difference? + +1. JS modules published to `@storacha` namespace on npm with the same names. _Exception_: `@web3-storage/w3up-client` renamed to just `@storacha/client`. +1. `blob/allocate` and `blob/accept` are no longer invoked on the service, they are invoked on an external service i.e. storage nodes. _Note_: they have actually been _renamed_ from `web3.storage/blob/allocate` and `web3.storage/blob/allocate`. +1. Removed the Store Protocol related code...it just doesn't work in a decentralized context and is deprecated anyway. _Exception_: I left the `store/*` capabilities defined in `@storacha/capabilities` (and the `web3.storage/*` ones for that matter). +1. Merged `w3cli` into the monorepo as `packages/cli` (and published on npm as `@storacha/cli`). +1. Deployed to `https://upload.storacha.network` (yes, unfortunately `https://up.storacha.network` is already taken), with service DID `did:web:upload.storacha.network`. _Note_: new service DID includes `upload.` i.e. it is NOT `did:web:storacha.network`. +1. Removed `allocations` table, added `blob-registry` table. Instead of recording allocations, we now "register" blobs. Registration is performed _after_ a blob is **accept**ed by a storage node. This gives us a more accurate view of what is stored in a space - mitigating the issue of a client failing to upload a blob after requesting to store it and still being billed for the used space. The `space/blob/list` invocation uses this table. +1. There is no Filecoin pipeline deployed and many `w3infra` stacks relating to legacy infra have been excluded from `upload-service-infra`. + +I believe these changes to be mainly _good_ for the project as a whole and aligned with where we want to get to but please leave comments to discuss if you have disagreement. + +### Why did you do this? + +* Easier, quicker and safer to fork and deploy a greenfield project (do not have to worry about affecting existing production deployment). +* Time pressure necessitated it. +* Fresh start, was able to shed some legacy baggage. +* As mentioned above, the changes are things we need to do eventually anyway IMHO. +* I wanted an as realistic as possible demo for our event in Bangkok. + +## Proposal + +We transition to working on `upload-service`, "forwardporting" any changes made in `w3up` that do not yet exist in `upload-service`. + +We _backport_ the relevant infra changes in `upload-service-infra` that were added to support "Routing Blob Protocol" to `w3infra`. + +* Change the `w3infra` service DID to `did:web:upload.storacha.network` and URL to `https://upload.storacha.network`. +* Point `https://up.web3.storage` and `https://up.storacha.network` to `https://upload.storacha.network`. +* Allow the service to receive invocations to audience `did:web:web.storage` and accept attestations signed by `did:web:web.storage`. +* Add the legacy Store Protocol handlers to `upload-service` by importing existing `@web3-storage/upload-api`. +* Setup and deploy a production storage provider node that writes to carpark. +* Migrate `allocation` table data to `blob-registry` table - this allows `space/blob/list` capability to work for older spaces. +* Build a maintenance mode switch to existing system allowing migration to take place without writes. +* In order to prevent confusion over what modules we're continuing to support, we will remove from the repos as much legacy code as possible. + * Import and re-export `@storacha` equivalents of all modules. Instead of simply deleting the packages this enables us to continue to easily make and track releases (bugfixes etc.) for the legacy packages whilst still removing the majority of the code. + * `w3up` repo changes: + * `access-client` + * import `@storacha/access` and re-export + * `blob-index` + * import `@storacha/blob-index` and re-export + * `capabilities` + * import `@storacha/capabilities` and re-export + * `did-mailto` + * import `@storacha/did-mailto` and re-export + * `eslint-config-w3up` + * leave as is (can remove when monorepo'd) + * `filecoin-api` + * import `@storacha/filecoin-api` and re-export + * `filecoin-client` + * import `@storacha/filecoin-client` and re-export + * `upload-api` + * Leave Store and Blob Protocol related handlers as is. + * Import `@storacha/upload-api` for all other handlers. This will remove most code from this package. + * `upload-client` + * import `@storacha/upload-client` and re-export + * `w3up-client` + * Import `@storacha/client` (and any other `@storacha/*` modules needed) and augment client with existing (but deprecated) Store Protocol methods. + * `w3infra` repo changes: + * import `@storacha/upload-api` + * `w3cli` repo changes: + * import `@storacha/cli` and re-export + * ...but first make `@storacha/cli` usable as library From 1f660a2b5abc34d022297110e1413fc6bec1c9c7 Mon Sep 17 00:00:00 2001 From: ash Date: Thu, 9 Jan 2025 16:14:55 +0000 Subject: [PATCH 2/5] forwardporting is just porting --- rfc/consolidating-w3up-and-upload-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/consolidating-w3up-and-upload-service.md b/rfc/consolidating-w3up-and-upload-service.md index 47ee0b5..7109d8d 100644 --- a/rfc/consolidating-w3up-and-upload-service.md +++ b/rfc/consolidating-w3up-and-upload-service.md @@ -30,7 +30,7 @@ I believe these changes to be mainly _good_ for the project as a whole and align ## Proposal -We transition to working on `upload-service`, "forwardporting" any changes made in `w3up` that do not yet exist in `upload-service`. +We transition to working on `upload-service`, porting any changes made in `w3up` that do not yet exist in `upload-service`. We _backport_ the relevant infra changes in `upload-service-infra` that were added to support "Routing Blob Protocol" to `w3infra`. From ec36d31df1b63a7be8799b6e758ea81bd95360e7 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 13 Jan 2025 11:39:59 +0000 Subject: [PATCH 3/5] fix: service DID --- rfc/consolidating-w3up-and-upload-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/consolidating-w3up-and-upload-service.md b/rfc/consolidating-w3up-and-upload-service.md index 7109d8d..5b5fbf3 100644 --- a/rfc/consolidating-w3up-and-upload-service.md +++ b/rfc/consolidating-w3up-and-upload-service.md @@ -36,7 +36,7 @@ We _backport_ the relevant infra changes in `upload-service-infra` that were add * Change the `w3infra` service DID to `did:web:upload.storacha.network` and URL to `https://upload.storacha.network`. * Point `https://up.web3.storage` and `https://up.storacha.network` to `https://upload.storacha.network`. -* Allow the service to receive invocations to audience `did:web:web.storage` and accept attestations signed by `did:web:web.storage`. +* Allow the service to receive invocations to audience `did:web:web3.storage` and accept attestations signed by `did:web:web3.storage`. * Add the legacy Store Protocol handlers to `upload-service` by importing existing `@web3-storage/upload-api`. * Setup and deploy a production storage provider node that writes to carpark. * Migrate `allocation` table data to `blob-registry` table - this allows `space/blob/list` capability to work for older spaces. From 043e69a417ab9d45df61a0765e1a796f8424f7ff Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 13 Jan 2025 11:40:22 +0000 Subject: [PATCH 4/5] feat: add detail for migration --- rfc/consolidating-w3up-and-upload-service.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/rfc/consolidating-w3up-and-upload-service.md b/rfc/consolidating-w3up-and-upload-service.md index 5b5fbf3..41e54d9 100644 --- a/rfc/consolidating-w3up-and-upload-service.md +++ b/rfc/consolidating-w3up-and-upload-service.md @@ -40,6 +40,19 @@ We _backport_ the relevant infra changes in `upload-service-infra` that were add * Add the legacy Store Protocol handlers to `upload-service` by importing existing `@web3-storage/upload-api`. * Setup and deploy a production storage provider node that writes to carpark. * Migrate `allocation` table data to `blob-registry` table - this allows `space/blob/list` capability to work for older spaces. + * This is a simple copy of all existing rows, with a couple of field renames (`multihash` -> `digest` and `invocation` -> `cause`). For reference, I've listed the current schemas: + * `allocation` schema: + * `space`: string (e.g. `did:key:space`) + * `multihash`: string (e.g. `zQm...`) + * `size`: number (e.g. `101`) + * `invocation`: string (e.g. `baf...ucan` - CID of invocation UCAN) + * `insertedAt`: string (e.g. `2022-12-24T...`) + * `blob-registry` schema: + * `space`: string + * `digest`: string + * `size`: number + * `cause`: string + * `insertedAt`: string * Build a maintenance mode switch to existing system allowing migration to take place without writes. * In order to prevent confusion over what modules we're continuing to support, we will remove from the repos as much legacy code as possible. * Import and re-export `@storacha` equivalents of all modules. Instead of simply deleting the packages this enables us to continue to easily make and track releases (bugfixes etc.) for the legacy packages whilst still removing the majority of the code. From eef73a2b4cf967d77e46e4f4d39ee85578ecbf9d Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Wed, 15 Jan 2025 11:02:51 +0000 Subject: [PATCH 5/5] refactor: apply feedback on migration strategy --- rfc/consolidating-w3up-and-upload-service.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/rfc/consolidating-w3up-and-upload-service.md b/rfc/consolidating-w3up-and-upload-service.md index 41e54d9..b4596d5 100644 --- a/rfc/consolidating-w3up-and-upload-service.md +++ b/rfc/consolidating-w3up-and-upload-service.md @@ -40,7 +40,7 @@ We _backport_ the relevant infra changes in `upload-service-infra` that were add * Add the legacy Store Protocol handlers to `upload-service` by importing existing `@web3-storage/upload-api`. * Setup and deploy a production storage provider node that writes to carpark. * Migrate `allocation` table data to `blob-registry` table - this allows `space/blob/list` capability to work for older spaces. - * This is a simple copy of all existing rows, with a couple of field renames (`multihash` -> `digest` and `invocation` -> `cause`). For reference, I've listed the current schemas: + * This is a table rename and a couple of field renames (`multihash` -> `digest` and `invocation` -> `cause`). For reference, I've listed the current schemas: * `allocation` schema: * `space`: string (e.g. `did:key:space`) * `multihash`: string (e.g. `zQm...`) @@ -53,7 +53,9 @@ We _backport_ the relevant infra changes in `upload-service-infra` that were add * `size`: number * `cause`: string * `insertedAt`: string -* Build a maintenance mode switch to existing system allowing migration to take place without writes. + * We will create the new table and _temporarily_ write to both. + * Once deployed we'll start the migration of older data to the new table. + * Once the migration is completed we'll remove the old table. * In order to prevent confusion over what modules we're continuing to support, we will remove from the repos as much legacy code as possible. * Import and re-export `@storacha` equivalents of all modules. Instead of simply deleting the packages this enables us to continue to easily make and track releases (bugfixes etc.) for the legacy packages whilst still removing the majority of the code. * `w3up` repo changes: