Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: change access-api wrangler.toml to be no_bundle=false #737

Closed
wants to merge 502 commits into from

Conversation

gobengo
Copy link
Contributor

@gobengo gobengo commented Apr 10, 2023

Motivation:

Experiment

  • I wanted to try some new ways of building and whether they would work, but without risking staging, so I used the 'dev' environment from wrangler.toml to try some deploys from local.
  • I made access-api use of wrangler more like toucan-js@3 wrangler-basic
  • I did a sample deploy from my laptop to 'dev' workers environment, with the sentry/toucan release called bengo-dev-0 sentry errors here
  • trigger error via GET /.debug/error
    • expectation: this to lead to an in-route error and show up in sentry with helpful stack traces - because this is what example implies should work
  • trigger error via route that uses d1 binding (e.g. ucanto routes triggered via w3up+ucanto observable pointing to dev env)
    • expectation: d1 error - because supposedly no_bundle=false (or omitting no_bundle) is incompatible with wrangler d1 bindings

Findings:

  • omg it works!
  • error triggered by GET /.debug/error (no d1)
  • error triggered by using d1 via access protocol via observable
    • The error did not happen in the access/authorize handler. That worked fine, and I got an email from dev env. The error happened when I clicked the email (invoking access/confirm), when the access/confirm invoacation handler tried to write to d1
    • It got a D1_ERROR, but I could see from the sentry report

      Error: ERROR 9009: SQL prepare error: no such table: delegations_v3

    • which makes sense, because dev doesn't have latest migrations applied that created that table. So I provisioned a new d1 database for this dev env, and ran npx wrangler --env=dev d1 migrations apply __D1_BETA__. This would error every couple migrations, but if I re-ran it it would make progress and error again (Internal error [code: 7501]), but eventually it would succeed at all migrations.
    • then I re-triggered things via the observable, get email, click email. I saw 'email validated'! 🎉
    • I continued using the observable to invoke access/claim (reads from D1), and got no error all the way through invoking from second device
      Screenshot 2023-04-10 at 5 48 41 PM

Next steps:

  • (opt) feedback on whether we can rely on this from @olizilla
  • develop this PR all the way to
    • make tests pass (tests rely on ./scripts/cli.js e.g. to run migrations, so I probably need to modify cli.js to still work in tests at least)
    • something that will work withour release automation. (locally I had to hardcode env.ACCOUNT_VERSION=bengo-dev-0 just to test the theory)
      • the toucan-js@3 examples hardcode 1.0.0 always. I think I can come up with something better than that quickly tomorrow

hugomrdias and others added 30 commits December 6, 2022 16:09
Motivation:
* latest commit on main branch was failing release workflow
[here](https://github.com/web3-storage/w3protocol/actions/runs/3621669639/jobs/6105393187#step:2:35)
* I think it's because the release-please config/manifest still referred
to `package/wallet`, which this PR removes
* seems like logical followup to
#248

Co-authored-by: Hugo Dias <hugomrdias@gmail.com>
Goals:
1. Decouple store from agent
2. Simplify agent creation
3. Agent governs data format not store
4. Initialization of agent data, not store
5. DRY initialization in agent, not repeated in each store impl

~~See:
https://gist.github.com/alanshaw/ea4bd2b0ab215ade696eac1300be577d~~
outdated

```js
// In regular use:
const store = new StoreIndexedDB()
const data = await store.load()
const agent = data
  ? Agent.from(data, { store })
  : await Agent.create({}, { store })

// Then StoreMemory can be deleted, since the AgentData already stores all data in
// memory. It's equivalent to:
const agent = await Agent.create()
```
🤖 I have created a release *beep* *boop*
---


##
[8.0.0](access-v7.0.2...access-v8.0.0)
(2022-12-06)


### ⚠ BREAKING CHANGES

* access-client store decoupling
([#228](#228))
* upgrade to `@ucanto/{interface,principal}`@^4.0.0
([#238](#238))
* follow up on the capabilities extract
([#239](#239))

### Features

* **access-client:** cli and recover
([#207](#207))
([720dafb](720dafb))
* follow up on the capabilities extract
([#239](#239))
([717fcaa](717fcaa))
* Revert "feat!: upgrade to `@ucanto/{interface,principal}`@^4.0.0"
([#245](#245))
([197439e](197439e))
* upgrade to `@ucanto/{interface,principal}`@^4.0.0
([#238](#238))
([309aff0](309aff0))


### Bug Fixes

* connection method is not async
([#222](#222))
([7c4e0b6](7c4e0b6))


### Code Refactoring

* access-client store decoupling
([#228](#228))
([64c7496](64c7496))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop*
---


##
[3.0.0](upload-client-v2.1.0...upload-client-v3.0.0)
(2022-12-06)


### ⚠ BREAKING CHANGES

* upload/list items have a shards array property
([#229](#229))
* upgrade to `@ucanto/{interface,principal}`@^4.0.0
([#238](#238))
* follow up on the capabilities extract
([#239](#239))

### Features

* **access-client:** cli and recover
([#207](#207))
([720dafb](720dafb))
* follow up on the capabilities extract
([#239](#239))
([717fcaa](717fcaa))
* Revert "feat!: upgrade to `@ucanto/{interface,principal}`@^4.0.0"
([#245](#245))
([197439e](197439e))
* upgrade to `@ucanto/{interface,principal}`@^4.0.0
([#238](#238))
([309aff0](309aff0))
* upload/list items have a shards array property
([#229](#229))
([723b281](723b281))


### Bug Fixes

* list response results return type
([#206](#206))
([e49c685](e49c685))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop*
---


##
[8.0.1](access-v8.0.0...access-v8.0.1)
(2022-12-07)


### Bug Fixes

* conf driver can store top level undefined
([5abc522](5abc522))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
…#246)

Motivation:
* second attempt at this
#238
* after first was reverted
#245

Warning:
* the versions of ucanto we're upgrading to were backward-incompatible.
storacha/ucanto#149
* Old 'SignerArchive' exports will probably no longer import using the
new `@ucanto/principal@^4.0.0` `SignerArchive`
* in slack, @alanshaw said he thinks this is unlikely to cause problems:
"There’s no code in production that uses toArchive"

Blockers
* [x] Wait for these two to land first (@hugomrdias says so)
  * [x] @hugomrdias #207
  * [x] @alanshaw #228
* [x] then will need to resolve merge conflicts on this branch
* [x] @gobengo resolve conflicts after
#207
* [x] @gobengo resolve conflicts after
#228

Co-authored-by: Irakli Gozalishvili <contact@gozala.io>
🤖 I have created a release *beep* *boop*
---


##
[9.0.0](access-v8.0.1...access-v9.0.0)
(2022-12-07)


### ⚠ BREAKING CHANGES

* upgrade access-api @ucanto/* and @ipld/dag-ucan major versions
([#246](#246))

### Features

* upgrade access-api @ucanto/* and @ipld/dag-ucan major versions
([#246](#246))
([65d191c](65d191c))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop*
---


##
[4.0.0](upload-client-v3.0.0...upload-client-v4.0.0)
(2022-12-07)


### ⚠ BREAKING CHANGES

* upgrade access-api @ucanto/* and @ipld/dag-ucan major versions
([#246](#246))

### Features

* upgrade access-api @ucanto/* and @ipld/dag-ucan major versions
([#246](#246))
([65d191c](65d191c))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop*
---


##
[5.0.0](upload-client-v4.0.0...upload-client-v5.0.0)
(2022-12-07)


### ⚠ BREAKING CHANGES

* upgrade access-api @ucanto/* and @ipld/dag-ucan major versions
([#246](#246))
* upload/list items have a shards array property
([#229](#229))
* upgrade to `@ucanto/{interface,principal}`@^4.0.0
([#238](#238))
* follow up on the capabilities extract
([#239](#239))
* rename callback
([#198](#198))

### Features

* [#153](#153)
([#177](#177))
([7fbf2a1](7fbf2a1))
* a base for a new web3.storage upload-client
([#141](#141))
([72c08d7](72c08d7))
* **access-client:** cli and recover
([#207](#207))
([720dafb](720dafb))
* account recover with email
([#149](#149))
([91ad47d](91ad47d))
* add static uploads API
([#148](#148))
([8b735fa](8b735fa))
* allow custom shard size
([#185](#185))
([2130405](2130405))
* explicit resource
([#181](#181))
([c127431](c127431))
* follow up on the capabilities extract
([#239](#239))
([717fcaa](717fcaa))
* Revert "feat!: upgrade to `@ucanto/{interface,principal}`@^4.0.0"
([#245](#245))
([197439e](197439e))
* support pagination
([#204](#204))
([363e3ac](363e3ac)),
closes [#201](#201)
* upgrade access-api @ucanto/* and @ipld/dag-ucan major versions
([#246](#246))
([65d191c](65d191c))
* upgrade to `@ucanto/{interface,principal}`@^4.0.0
([#238](#238))
([309aff0](309aff0))
* upload/list items have a shards array property
([#229](#229))
([723b281](723b281))


### Bug Fixes

* export types
([c44a252](c44a252))
* list response results return type
([#206](#206))
([e49c685](e49c685))
* store export
([#197](#197))
([d2296c5](d2296c5))
* upload client list needs empty nb
([#194](#194))
([7894fb9](7894fb9))
* uploadFile requires BlobLike not Blob
([56ec496](56ec496))


### Code Refactoring

* rename callback
([#198](#198))
([2349763](2349763))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop*
---


##
[2.0.0](capabilities-v1.0.0...capabilities-v2.0.0)
(2022-12-07)


### ⚠ BREAKING CHANGES

* upgrade access-api @ucanto/* and @ipld/dag-ucan major versions
([#246](#246))
* upgrade to `@ucanto/{interface,principal}`@^4.0.0
([#238](#238))
* follow up on the capabilities extract
([#239](#239))

### Features

* **access-client:** cli and recover
([#207](#207))
([720dafb](720dafb))
* follow up on the capabilities extract
([#239](#239))
([717fcaa](717fcaa))
* Revert "feat!: upgrade to `@ucanto/{interface,principal}`@^4.0.0"
([#245](#245))
([197439e](197439e))
* upgrade access-api @ucanto/* and @ipld/dag-ucan major versions
([#246](#246))
([65d191c](65d191c))
* upgrade to `@ucanto/{interface,principal}`@^4.0.0
([#238](#238))
([309aff0](309aff0))


### Bug Fixes

* fix Access API cannot get space/info
[#243](#243)
([#255](#255))
([1a74031](1a74031))
* generated typdefs
([#258](#258))
([dfd2bd5](dfd2bd5))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: Benjamin Goering <171782+gobengo@users.noreply.github.com>
Cannot put `undefined` in UCANs
🤖 I have created a release *beep* *boop*
---


##
[9.0.1](access-v9.0.0...access-v9.0.1)
(2022-12-08)


### Bug Fixes

* validate agent name
([#271](#271))
([5a1fdb2](5a1fdb2))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
I noticed earlier today that running `pnpm run docs` fails with a bunch
of typescript errors unless you run `pnpm run build` first.

This just updates the `docs` npm script to `pnpm run build && typedoc
--out docs`, so we'll always have build output to make typedoc happy.
Buffer has a toJSON implementation which is called before our replacer
function when serialising. It's JSON form is an object like `{ type:
'Buffer, data: number[] }`.

This means it doesn't get identfied as $bytes, and does not get
deserialised in the reviver. I think we'd rather have `Uint8Array`
everywhere so this PR fixes it so that it gets revived as that.

License: MIT
Signed-off-by: Oli Evans <oli@protocol.ai>
…r id (#275)

Motivation:
* #237

Notes
* This is intended to be safe to merge/deploy as is. The new
functionality can be opted-into via env vars
* The ucanto server will only be constructed with an opts.id that is a
signer `withDID(...)` iff `process.env.DID` is set to a DID
* If `process.env.DID` is unset, the ucanto server id signer will have a
`did()` corresponding to `config.PRIVATE_KEY` (it will be a `did:key` of
the corresponding public key)
* Before this PR, many tests of the access-api worker were configured
implicitly via dotenv with whatever the `/.env.local` file happened to
contain. I added a worker test that asserts functionality that is
dependent on the details of the env vars, so I made an affordance for
[tests to be able to configure the access-api worker with environment
variables defined in the test case
itself](https://github.com/web3-storage/w3protocol/pull/275/files#diff-e20ffc550d006747790e7b67da280446c1cd1d2d4f72ccb5df04f62cbb6423daR149).
- add methods toBytes and fromBytes useful to store as bytes instead of
strings
- `encodeDelegations` is now `delegationsToString``
- `decodeDelegations` is now `stringToDelegations``
- all functions are sync now

Writing delegations to a car uses `@ipld/car/buffer-writer` now to make
it sync, reading from a CAR it's more complicated 🤫.
`ipld/car` does not support sync reading so this PR includes a refactor
of `CarReader` to be sync, this is a temporary solution until i can get
a proper PR to `ipld/car` to add support for sync reading. It's not
trivial to do so because of the way ipld/car code is structured and i
dont want to just add a sync copy/paste of the decoder functions to it.
The release workflow wasnt running the docs job because of an `if`
clause leftover from the manual workflow.
By removing the side effect many methods that read data in memory no
longer have to be async. I think this makes the API simplier.

I've added a _method_ that can be called to clean up expired
delegations, which shifts the responsibility onto the user.
Alternatively we could move the clean up into the methods that write
delegations? Open to other suggestions.

resolves #219
needed by storacha/w3cli#20 to find which
shards to also remove when removing an upload

License: MIT
Signed-off-by: Oli Evans <oli@protocol.ai>
🤖 I have created a release *beep* *boop*
---


##
[5.1.0](upload-client-v5.0.0...upload-client-v5.1.0)
(2022-12-13)


### Features

* sync encode/decode delegations
([#276](#276))
([9d48372](9d48372))
* upload/remove returns removed item
([#289](#289))
([472d6b6](472d6b6))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
- spaces.metadata is now nullable
- d1 tables are now typed
- moved from `workers-qb` to
[`kysely`](https://github.com/koskimas/kysely) with [d1
dialect](https://github.com/aidenwallis/kysely-d1)


closes #280
🤖 I have created a release *beep* *boop*
---


##
[9.1.0](access-v9.0.1...access-v9.1.0)
(2022-12-13)


### Features

* sync encode/decode delegations
([#276](#276))
([9d48372](9d48372))


### Bug Fixes

* destructured save
([#272](#272))
([b6aaf9b](b6aaf9b))
* handle Buffer serialization
([#277](#277))
([8e8f745](8e8f745))
* make d1 spaces.metadata nullable and change to kysely
([#284](#284))
([7f09479](7f09479)),
closes [#280](#280)

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
This adds an About section with some context about the access-client and
how it fits into the bigger picture.

It also adds some detail to the Usage section, which now has subsections
for creating Agents & Spaces and registering Spaces with the access
service.

Closes #251
🤖 I have created a release *beep* *boop*
---


##
[2.1.0](capabilities-v2.0.0...capabilities-v2.1.0)
(2022-12-13)


### Features

* sync encode/decode delegations
([#276](#276))
([9d48372](9d48372))


### Bug Fixes

* make d1 spaces.metadata nullable and change to kysely
([#284](#284))
([7f09479](7f09479)),
closes [#280](#280)

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Travis Vachon and others added 13 commits April 5, 2023 12:00
…dProgress callback (#720)

Use the ipfs-utils fetch implementation to enable upload progress
reporting.

Users of `upload-client` can now pass an `onUploadProgress` callback to
`uploadFile` and `uploadDirectory` which will be handed off to the
`fetch` polyfill from `ipfs-utils`. We wrap `onUploadProgress` in order
to set `url` on the progress status events we get from `fetch` so that
clients can differentiate between upload progress events from different
shards, which will be interleaved.
🤖 I have created a release *beep* *boop*
---


##
[8.3.0](upload-client-v8.2.0...upload-client-v8.3.0)
(2023-04-05)


### Features

* allow clients to pass uploadFile and uploadDirectory an
onUploadProgress callback
([#720](#720))
([20bc69f](20bc69f))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Motivation:
* #623 
* test theory that things will work if we upgrade to toucan-js@3, which
is a rewrite that sends source maps in a new way
🤖 I have created a release *beep* *boop*
---


## 1.0.0 (2023-04-06)


### ⚠ BREAKING CHANGES

* add did mailto package, replacing `createDidMailtoFromEmail`
([#722](#722))

### Features

* add did mailto package, replacing `createDidMailtoFromEmail`
([#722](#722))
([5a4123f](5a4123f))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop*
---


##
[12.0.0](access-v11.2.0...access-v12.0.0)
(2023-04-06)


### ⚠ BREAKING CHANGES

* add did mailto package, replacing `createDidMailtoFromEmail`
([#722](#722))

### Features

* add did mailto package, replacing `createDidMailtoFromEmail`
([#722](#722))
([5a4123f](5a4123f))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop*
---


##
[5.5.1](w3up-client-v5.5.0...w3up-client-v5.5.1)
(2023-04-06)


### Bug Fixes

* add newline to space.js to test release-please 694 fix
([#696](#696))
([6bc4dc3](6bc4dc3))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>
🤖 I have created a release *beep* *boop*
---


##
[6.0.0](access-api-v5.2.1...access-api-v6.0.0)
(2023-04-06)


### ⚠ BREAKING CHANGES

* add did mailto package, replacing `createDidMailtoFromEmail`
([#722](#722))

### Features

* add did mailto package, replacing `createDidMailtoFromEmail`
([#722](#722))
([5a4123f](5a4123f))


### Bug Fixes

* bug where did:mailto with pct-encoded email parts was not decoded
correctly ([#719](#719))
([c367639](c367639))
* upgrade toucan-js and @web3-storage/worker-utils
([#728](#728))
([853559d](853559d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: Benjamin Goering <171782+gobengo@users.noreply.github.com>
)

Motivation:
*
#623 (comment)
* this was suggested by toucan-js author as no longer being needed in
3.0.0.
Motivation:
* #623 - see if errors from this show up different than errors from
ucanto HandlerExceptionError
…tely-defined object in hopes stack traces will work better (#733)

Motivation:
* #623
@gobengo gobengo changed the title change access-api wrangler.toml to be no_bundle=false feat: change access-api wrangler.toml to be no_bundle=false Apr 11, 2023
@gobengo gobengo requested a review from olizilla April 11, 2023 00:52
@gobengo gobengo marked this pull request as ready for review April 11, 2023 00:52
@@ -23,20 +17,22 @@ preview_id = "62e57652625c44a3b1fef2f840ffc882"
[[d1_databases]]
binding = "__D1_BETA__"
database_name = "spaces-dev"
database_id = "7c676e0c-b9e7-4711-97c8-7b1c8eb229ae"
database_id = "e3331d3b-d8dd-48b0-ab80-2dba93c2c256"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new db because the old one is quite out of date, has no important data, and got hung up on migrations. No one has been using dev afaik so I think this should be safe.


[[r2_buckets]]
binding = "DELEGATIONS_BUCKET"
bucket_name = "w3up-delegations-dev-0"
preview_bucket_name = "w3up-delegations-dev-0"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrangler asked me to do this when deploy to dev env


[vars]
ENV = "dev"
DEBUG = "true"
DID = "did:web:local.web3.storage"
PRIVATE_KEY="MgCYWjE6vp0cn3amPan2xPO+f6EZ3I+KwuN1w2vx57vpJ9O0Bn4ci4jn8itwc121ujm7lDHkCW24LuKfZwIdmsifVysY="
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is from ../../.env.tpl so already public

@gobengo
Copy link
Contributor Author

gobengo commented Apr 11, 2023

I can't think of any downsides. It would accomplish:

  • working stack traces in sentry
  • access-api use of cloudflare workers is much more like what nftstorage/website and web3-storage/website use

downside

  • this uses d1 + no_bundle=false - which is technically not allowed in d1 public alpha, but seems like maybe we are detecting it now works even though they haven't updated docs like https://developers.cloudflare.com/workers/wrangler/bundling/#d1-bindings

Sanity check

@gobengo
Copy link
Contributor Author

gobengo commented Apr 11, 2023

I force pushed to this source branch and deleted it because I accidentally pushed a commit containing a sentry dsn I used to test. I disabled the DSN in sentry admin, so it's now useless. I followed these docs to delete using git-filter-repo.

It should be impossible to find the accidentally pushed DSN, and if it is found, it should be useless

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.