access-api stacktraces in sentry are not working

[vasco-santos]

No description provided.

[gobengo]

related: #367
tl;dr - we are a bit constrained in how we do this due to d1 public alpha 'no-bundle'.

these starters from toucan-js seem like a promising direction. https://github.com/robertcepa/toucan-js#examples

[gobengo]

As long as we use d1 which requires no_bundle=false (or omitted) in wrangler.toml, I think we'd have to do something like this which pushes source maps to sentry via sentry-cli in our deploy pipeline. https://github.com/robertcepa/toucan-js/tree/master/examples/wrangler-basic#wrangler-basic-starter

After removing d1, we may be able to go back to no_bundle=true and use some of the other setups if desired. (but also maybe 'basic' is fine)

[gobengo]

I just learned some things:

• there is already some support in the access-api build script to upload the source maps to sentry: https://github.com/web3-storage/w3up/blob/main/packages/access-api/scripts/cli.js#L77
• afaict, process.env.SENTRY_UPLOAD === 'true' is true in the github actions that deploy to staging and production. e.g. this sample deploy seems to have created a release in sentry
• I see source maps for recent releases in staging and production in the source maps settings for access-api in our sentry
• I see releases in sentry from latest deploys
• and recent errors seem to show like release: 5.2.1-production (a4d3409)
  • So I was surprised that error still doesn't have very much help from source maps.

I noticed this issue #254
mentions upgrading toucan-js, which is the sentry library that access-api uses.

• that 3.0.0 release

  This is a complete rewrite of toucan-js. The goal of this update is to reuse more components from @sentry/core, fix long-standing issues with source maps, and provide starters using various bundlers (esbuild, rollup, vite, webpack).

• mentions source maps, seems worth trying
• rn access-api uses toucan-js ^2.7.0, so we aren't yet using the latest and greatest source maps stuff.

So what I will try is upgrading toucan-js to >= 3.x, as suggested by #254

[gobengo]

• need to upgrade @web3-storage/worker-utils toucan version as well upgrade toucan to 3.x workers#20

[gobengo]

• I deployed the toucan-js@3.x upgrade to access-api staging env
• triggered error on staging using w3cli W3_UPLOAD_SERVICE_DID='did:web:staging.web3.storage' W3_UPLOAD_SERVICE_URL='https://staging.up.web3.storage/' W3_ACCESS_SERVICE_DID='did:web:staging.web3.storage' W3_ACCESS_SERVICE_URL='https://w3access-staging.protocol-labs.workers.dev' DEBUG='*' ./shim.js authorize "example@example.com"
  • example@example.com leads to an error (useful here)
  • error in sentry

    • no stack traces :/ (beyond showing export default worker like before)

    • but I also see a 'Tell us where your source code is' link on that. It asked me to give it a github url to '../src/index.js', so I did https://github.com/web3-storage/w3up/blob/main/packages/access-api/src/index.js

    • sentry didn't know about the w3up repo, so I added it https://protocol-labs-it.sentry.io/settings/integrations/github/109727/

    • It made 'open this line with github' work, but it still just takes you to the line for export default worker

[gobengo]

I've been trying to figure out what to try next, and just noticed:

• the sentry error still doesn't have a source-map aware stack trace
• says 6.0.0-staging (99c532a) url
  • in sentry, that release exists and does have 2 source map artifacts
• what does that sha 99c532a in the release correspond to?
  • if I make a github url for it, something loads 99c532a but it says "This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository."
• theory:
  • the staging release happens from branch release-please--branches--main--components--access-api managed by release-please-action
  • It appears release-please action, doesn't append new commits to that branch, it force-pushes a new one and removes the old one
  • that's unfortunate, and I'm tempted to fix, but that alone wouldn't explain the issue afaict

[gobengo]

oooh I should try what @olizilla did here
https://github.com/web3-storage/web3.storage/pull/1033/files#diff-fcc5ef774a5ba3602a1c3478168015fa7b4b4b154887b7d2ed197720f65501f1R57

(It's funny, I was reading this toucan-js issue thread and organically stumbled upon an @olizilla robertcepa/toucan-js#54 (comment) )

[gobengo]

I should also try not using rewriteFrames as suggested robertcepa/toucan-js#54 (comment)

• ✅ tried it, but it didn't fix with this error

I did just notice that this error, and the last one too, show more info if I click '...' and then check the 'minified' checkbox on the stack trace. Then I can see that this error is coming from inside a fetch invocaiton.
That makes me think that

• toucan-js is sending good information about where the errors are coming from
• but the source map files aren't helping map back to the unminified source
  • which is the type of thing the other change might help with access-api stacktraces in sentry are not working #623 (comment)

[gobengo]

I wondered if any part of it was due to the way I was triggering the error that got into sentry, which was a ucanto HandlerExecutionError. (I was thinking maybe someone needed to call Error.captureStackTrace(), I don't think that's needed the way ucanto subclasses Error).

So I added a simple route that will throw an error (as oli had done while testing this type of thing).
That produced this error but did not affect stack trace rendering. :|

minified raw stack trace

Error: this is the result of handling a request at /.debug/error
  at <anonymous>(worker.js:20855:9)
  at Yd.fetch(worker.js:8599:23)
  at Object.fetch(worker.js:20861:22)
  at Object.fetch(worker.js:21072:16)

If I uncheck 'minified'

Error: this is the result of handling a request at /.debug/error
  at <unknown>(../src/index.js:44:16)
  at <unknown>(../src/index.js:44:16)
  at <unknown>(../src/index.js:44:16)
  at <unknown>(../src/index.js:44:16)

Why would this be? (!!!)

Brainstorms (I will think over weekend and ask for advice from @olizilla et al)

• Is there something about how the entrypoint does export default worker that's messing it up? How do our other workers projects with functional sentry stack traces do it?
  • nftstorage uses the old addEventListener('fetch') syntax https://github.com/nftstorage/nft.storage/blob/main/packages/api/src/index.js#L252
  • toucan-js example wrangler-esbuild does it with inline object https://github.com/robertcepa/toucan-js/blob/master/examples/wrangler-esbuild/src/index.ts#L8
  • web3.storage/website uses an inline object with fetch method https://github.com/web3-storage/web3.storage/blob/main/packages/api/src/index.js#L161
    • it's worth trying this way on access-api. it's easy. and it's what the toucan-js example does

Other things worth trying:

• use @sentry/esbuild-plugin like the toucan-js examples do instead of what access-api is doing: https://github.com/robertcepa/toucan-js/blob/master/examples/wrangler-esbuild/esbuild.config.js#L15
  • I don't think this should be needed, because web3.storage/website and nftstorage/website dont do it, and they do something more like what access-api is doing now (using sentry-cli as a library) https://github.com/nftstorage/nft.storage/blob/main/packages/api/scripts/cli.js#L79

[gobengo]

• meet with @olizilla to brainstorm which other things to try

• investigate why sentry error still shows line export default worker even though after my change last friday I'd expect it to be like export default { or show the fetch method on that obj

  • I needed to generate a new error on staging using the change I had deployed. Then it still doesn't show a helpful stack trace, it just shows the end of the inlined exported object.

• inspect the source map files being generated and sent to sentry

  • I did this a bit with a tool like (I forget which) sourcemaps.io last week. They seemed to work just fine, but I haven't used that tool before, so I should try again and document what I did
  • I retested using this tool and js+js.map from latest staging release. No errors reported.

--

Ideas

• try using @sentry/esbuild-plugin to send the source maps https://github.com/robertcepa/toucan-js/blob/master/examples/wrangler-esbuild/esbuild.config.js#L15
  • ⚠️ after talking with @olizilla I realized this probably won't help. The underlying problem is that no_bundle=false means that cloudflare bundling is probably generating a bad source map
  • pro: it's what toucan-js 3.0.0 examples use, so it's likely to get the most support from @robertcepa if we followup on this past help
  • con: it's not what our other services do (afiact), so it's making things less like our other services.
    • Though those other services use toucan-js 2.7.0 anyway. so if we scout out a way that works with toucan-js 3.0.0, maybe those other services would benefit

[gobengo]

• try not sending source maps as part of CI, and just using the source maps generated by cloudflare bundle (no_bundle=false) which seems same as wrangler-basic starter
  • set staging env var SENTRY_UPLOAD=0 (!== "true") to test, re-ran deploy staging job
• ask for help from toucan-js author where they last made a good suggestion