Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: merge @web3-storage/access-api into @web3-storage/upload-api #790

Merged
merged 55 commits into from
Jun 7, 2023

Conversation

travis
Copy link
Member

@travis travis commented May 11, 2023

image

As part of the transition from D1 to Dynamo for delegation and provision storage, move service providers from @web3-storage/access-api to @web3-storage/upload-api.

This change decouples this entire repository from any particular storage implementation and should make it easier for third parties to create w3up-compatible storage services. We now expose 2 storage interfaces - DelegationsStorage and ProvisionsStorage - and an email provider interface (right now simply called Email) that implementors like our very own w3infra need to implement. A prototype implementation of these interfaces can be found in storacha/w3infra#194 - feedback on that PR was the basis for some of the public interface changes in this PR.

I've also taken this opportunity to drop support for the voucher protocol and stop using the spaces table which appears to only support that. I've also dropped the accounts table since it didn't seem to be adding anything - the one query we made to it returned the account DID that we used to query the table. Once we have a clearer picture of what information we want to store in an account we can bring that back if that makes sense.

Important notes for reviewers:

  • There are definitely some awkward interfaces hanging around after merging these two codebases - please feel free to call them out, and I'll either fix them now or file some fast follow issues for more time-intensive refactors.
  • I've deleted access-api primarily because it makes the PR much more readable. I'm open to removing it with this PR, but also open to keeping it in - if we decide to keep it for now, I'll restore it just before merging.
  • @Gozala has identified a number of improvements to the interfaces and underlying design - some of them I hope to incorporate into this round of work and may result in some (hopefully small) changes to this PR, some of which I think make sense for future work
  • we should rename upload-api. should we rename upload-api? probably? now or later? I don't have super strong feelings, but we should probably do this soonish?

TODO

@travis travis changed the title wip: move access api providers to upload-api feat!: move access api providers to upload-api May 15, 2023
@travis travis marked this pull request as ready for review May 16, 2023 01:33
if we don't have a customer with a particular DID, return null

also get provision and delegation tests actually running!
travis added 2 commits May 31, 2023 16:49
I don't think a `Customer.get` should return a customer if we haven't provisioned anything for them, so remove that check from this test
@travis
Copy link
Member Author

travis commented Jun 1, 2023

woo hoo! updated per all of these comments.

Only thing I would like to be addressed is the fact that we simply return customer DID from the customer/has without any checks.

I've done this in 4287e85, but it's induced a bit of a change that broke tests in a way that I fixed in 35cff74

The idea behind the change is that we shouldn't consider a user a "customer" until we've provisioned storage for them. The test I updated in 35cff74 would verify that after a user did an access/authorize we'd get a customer record back when we invoked customer/get with their DID.

This is tricky the way things are currently implemented because we don't actually save a "subscription" record until we allocate that customer storage, and that's what we currently use to track customers.

We could introduce a new "customers" table and add a record to it when we invoke access/authorize but that's actually not my preference - I'd like to avoid tracking information about customers unless we have a relationship with them that means we'll be storing non-delegation data for them, and I like how the current system operates in that regard.

@Gozala what do you think? Happy to jump on a quick call tomorrow to chat more if that'd be helpful!

@travis travis requested review from Gozala and vasco-santos June 1, 2023 00:23
@travis
Copy link
Member Author

travis commented Jun 1, 2023

Also @Gozala just want to make sure that 4287e85 addressed the issue you mentioned above with customer/get - if it does could you hit the approve button?

travis added 6 commits June 1, 2023 11:33
we had some tests that said a space couldn't have two providers and some that said they could. my understanding is that we do not want to support multiple providers on a space, so update tests accordingly.

these changes flowed out of the dynamodb implementation and things I learned while writing that and generally make our mock provisions storage more consistent with that implementation
Travis Vachon and others added 4 commits June 6, 2023 14:46
Move the `cause` parameter of `putMany` into an options hash and make
clear that it is an optional parameter intended to allow some storage
implementations to not store delegations themselves but instead rely on
CID-identified UCAN invocations that they already have access to.
@travis travis merged commit 4f6ddb6 into main Jun 7, 2023
@travis travis deleted the feat/export-access-api-handlers branch June 7, 2023 22:31
travis pushed a commit that referenced this pull request Jun 7, 2023
🤖 I have created a release *beep* *boop*
---


##
[6.0.0](capabilities-v5.0.1...capabilities-v6.0.0)
(2023-06-07)


### ⚠ BREAKING CHANGES

* merge `@web3-storage/access-api` into `@web3-storage/upload-api`
([#790](#790))

### Features

* merge `@web3-storage/access-api` into `@web3-storage/upload-api`
([#790](#790))
([4f6ddb6](4f6ddb6))
* w3 aggregate protocol client and api implementation
([#787](#787))
([b58069d](b58069d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
travis pushed a commit that referenced this pull request Jun 8, 2023
🤖 I have created a release *beep* *boop*
---


##
[14.0.0](access-v13.0.2...access-v14.0.0)
(2023-06-07)


### ⚠ BREAKING CHANGES

* merge `@web3-storage/access-api` into `@web3-storage/upload-api`
([#790](#790))

### Features

* merge `@web3-storage/access-api` into `@web3-storage/upload-api`
([#790](#790))
([4f6ddb6](4f6ddb6))
* w3 aggregate protocol client and api implementation
([#787](#787))
([b58069d](b58069d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>
travis pushed a commit that referenced this pull request Jun 8, 2023
🤖 I have created a release *beep* *boop*
---


##
[4.0.0](upload-api-v3.0.0...upload-api-v4.0.0)
(2023-06-08)


### ⚠ BREAKING CHANGES

* merge `@web3-storage/access-api` into `@web3-storage/upload-api`
([#790](#790))

### Features

* merge `@web3-storage/access-api` into `@web3-storage/upload-api`
([#790](#790))
([4f6ddb6](4f6ddb6))


### Bug Fixes

* upgrade remaining ucanto deps
([#798](#798))
([7211501](7211501))
* upgrade ucanto to 8
([#794](#794))
([00b011d](00b011d))
* use legacy codec on upload api
([#788](#788))
([1514474](1514474))
* use legacy codec on upload api
([#788](#788))
([84a4d44](84a4d44))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
travis pushed a commit that referenced this pull request Jun 8, 2023
🤖 I have created a release *beep* *boop*
---


##
[7.0.0](w3up-client-v6.0.1...w3up-client-v7.0.0)
(2023-06-08)


### ⚠ BREAKING CHANGES

* merge `@web3-storage/access-api` into `@web3-storage/upload-api`
([#790](#790))

### Features

* merge `@web3-storage/access-api` into `@web3-storage/upload-api`
([#790](#790))
([4f6ddb6](4f6ddb6))
* w3 aggregate protocol client and api implementation
([#787](#787))
([b58069d](b58069d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>
travis pushed a commit to storacha/w3infra that referenced this pull request Jul 18, 2023
Get `@web3-storage/access-api` handlers running in the `upload-api`
service.

This introduces two new DynamoDB tables to store "delegations" and
"provisions". NB: I have a related PR in
#200 that uses "consumers"
and "subscriptions" tables rather than "provisions".

This also adds a new Lambda to handle the email verification flow - I've
brought some HTML utilities over from `@web3-storage/access-api` for
this purpose and wired the handlers up to the same path they were at in
the old service `/validate-email`.

This is the part 2 of a larger project (started in
storacha/w3up#790) to integrate the access
service into the w3infra upload service, which will let us move
operational details out of w3up, retire D1, consolidate our operations
around AWS Lambdas, deprecate now-unused functionality like the voucher
flows and generally streamline our UCAN handlers and the infrastructure
that supports them.

---------

Co-authored-by: Irakli Gozalishvili <contact@gozala.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants