Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: change access-api wrangler.toml to be no_bundle=false. use wrangler bundling #739

Merged
merged 25 commits into from
Apr 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
1ffb681
change access-api wrangler.toml to be no_bundle=false. use wrangler b…
gobengo Apr 10, 2023
7cf6278
prettier
gobengo Apr 11, 2023
4ec1075
rm build step from access-api envs dev,staging,production (use built-…
gobengo Apr 13, 2023
53ff41f
create new dev d1 db database_id for name access-dev (cloudflare depl…
gobengo Apr 13, 2023
92a89d3
make sentry-release-version code a reusable function/script
gobengo Apr 13, 2023
9570179
wip release scripts
gobengo Apr 13, 2023
6927148
wip
gobengo Apr 13, 2023
27eff7b
wip
gobengo Apr 13, 2023
8465228
add release name prep to reusable-deploy-api
gobengo Apr 13, 2023
df8d799
prettier
gobengo Apr 13, 2023
87bd214
fix how reusable-deploy-api gets release name
gobengo Apr 13, 2023
9cb44c3
reusable-deploy-api only installs packages on dep tree of access-api
gobengo Apr 13, 2023
ba143bd
reusable-deploy-api publishes sentry release and source maps
gobengo Apr 13, 2023
f3446a0
wip
gobengo Apr 13, 2023
50ded92
wip
gobengo Apr 13, 2023
25aecb0
wip
gobengo Apr 14, 2023
6f16f09
wip
gobengo Apr 14, 2023
fb13a84
wip
gobengo Apr 14, 2023
0be6257
remove extra quotes
gobengo Apr 14, 2023
19b7cd3
clean up reusable-deploy-api.yml
gobengo Apr 14, 2023
3cecdf6
typo
gobengo Apr 14, 2023
7aabcc0
remove console
gobengo Apr 14, 2023
1d7f23e
access-api wranger.toml doesn't specify rules unnecessarily
gobengo Apr 14, 2023
3af1c35
try only running wrangler publish once
gobengo Apr 14, 2023
d58f3b6
dont pass unnecessary env vars to wrangler publish
gobengo Apr 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 32 additions & 4 deletions .github/workflows/reusable-deploy-api.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,23 @@ jobs:
with:
node-version: 18
cache: 'pnpm'
- run: pnpm install
# Migration database
- run: pnpm --filter '@web3-storage/access-api...' install
# get release name that will be used for sentry
- id: set-release-name
env:
ENV: ${{ inputs.environment }}
run: |
echo "release_name=$(node packages/access-api/scripts/release.js)" >> "$GITHUB_OUTPUT"
- run: echo "release name is ${{ steps.set-release-name.outputs.release_name }}"
# write release info to ./src so it can be imported at runtime (e.g. used by toucan-js for `opts.release`)
# be sure to keep this before wrangler-action bundles+deploys
- name: write release info to src
working-directory: packages/access-api/
env:
ENV: ${{ vars.ENV }}
run: |
node scripts/release.js esm > src/utils/release.build.js
# Apply D1 Migrations
- run: pnpm -r --filter @web3-storage/access-api exec wrangler d1 migrations apply __D1_BETA__ --env ${{ inputs.environment }}
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CF_TOKEN }}
Expand All @@ -53,6 +68,7 @@ jobs:
with:
# preCommands: git config --global --add safe.directory "*"
apiToken: ${{ secrets.CF_TOKEN }}
command: publish --env "${{ vars.ENV }}" --outdir=dist
workingDirectory: 'packages/access-api'
environment: ${{ inputs.environment }}
secrets: |
Expand All @@ -65,7 +81,19 @@ jobs:
POSTMARK_TOKEN: ${{ secrets.POSTMARK_TOKEN }}
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
SENTRY_UPLOAD: ${{ secrets.SENTRY_UPLOAD }}
SENTRY_TOKEN: ${{ secrets.SENTRY_TOKEN }}
LOGTAIL_TOKEN: ${{ secrets.LOGTAIL_TOKEN }}
UCAN_LOG_BASIC_AUTH: ${{ secrets.UCAN_LOG_BASIC_AUTH }}
- name: create sentry release
working-directory: packages/access-api
run: |
ls -alh ./dist
# create sentry release
pnpm exec sentry-cli releases new "$RELEASE_NAME" --finalize
# associate wrangler-built src+map to sentry release
pnpm exec sentry-cli releases files "$RELEASE_NAME" upload-sourcemaps ./dist
env:
ENV: ${{ vars.ENV }}
RELEASE_NAME: ${{ steps.set-release-name.outputs.release_name }}
SENTRY_ORG: ${{ vars.SENTRY_ORG }}
SENTRY_PROJECT: ${{ vars.SENTRY_PROJECT }}
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_TOKEN }}
1 change: 1 addition & 0 deletions packages/access-api/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.dev.vars
52 changes: 52 additions & 0 deletions packages/access-api/scripts/release.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
/* eslint-disable unicorn/prefer-top-level-await */
/* eslint-disable no-console */
/* eslint-disable unicorn/prefer-module */
/**
* @file
* use this at build time (node.js) to create files that can be
* imported at runtime (maybe workerd).
*/
import sade from 'sade'
import * as url from 'node:url'
import { getReleaseName } from '../src/utils/release.node.js'
import path from 'path'
import { fileURLToPath } from 'url'
// @ts-ignore
import git from 'git-rev-sync'

function __dirname() {
return path.dirname(fileURLToPath(import.meta.url))
}

if (import.meta.url.startsWith('file:')) {
const modulePath = url.fileURLToPath(import.meta.url)
if (process.argv[1] === modulePath) {
main().catch((error) => {
throw error
})
}
}

async function main(argv = process.argv) {
const cli = sade('access-api-release')
cli
.command('name', '', { default: true })
.option('--env', 'Environment', process.env.ENV)
.action((opts) => {
const releaseName = getReleaseName(opts.env)
console.log(releaseName)
})
cli
.command('esm', 'print release info as ES Module string')
.option('--env', 'Environment', process.env.ENV)
.action((opts) => {
const lines = [
`/** @type {string|undefined} */`,
`export const gitRevShort = '${git.short(__dirname())}'`,
`/** @type {string|undefined} */`,
`export const name = '${getReleaseName(opts.env)}'`,
]
console.log(lines.join('\n'))
})
cli.parse(argv)
}
Comment on lines +30 to +52
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is neat, but note that our other projects do this by injecting these in as variables into the bundle at build time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They do that using no_bundle=true and injection via custom esbuild, so this is a way of doing it with no_bundle=false.
Yes it's different, but it seems worth it in order to get the stack traces.

If there's another way that works that seems better, lmk and I can try to implement.

21 changes: 7 additions & 14 deletions packages/access-api/src/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
// eslint-disable-next-line no-unused-vars
import * as UCAN from '@ucanto/interface'
import { DID } from '@ucanto/core'
import * as release from './utils/release.build.js'

/**
* Loads configuration variables from the global environment and returns a JS object
Expand All @@ -14,15 +15,7 @@ export function loadConfig(env) {
const vars = {}

/** @type {Array<keyof env>} */
const required = [
'DID',
'ENV',
'DEBUG',
'PRIVATE_KEY',
'SENTRY_DSN',
'POSTMARK_TOKEN',
'LOGTAIL_TOKEN',
]
const required = ['DID', 'ENV', 'DEBUG', 'PRIVATE_KEY']

for (const name of required) {
const val = env[name]
Expand All @@ -47,21 +40,21 @@ export function loadConfig(env) {

POSTMARK_TOKEN: vars.POSTMARK_TOKEN,
POSTMARK_SENDER: env.POSTMARK_SENDER,
SENTRY_DSN: vars.SENTRY_DSN,
LOGTAIL_TOKEN: vars.LOGTAIL_TOKEN,
SENTRY_DSN: env.SENTRY_DSN,
LOGTAIL_TOKEN: env.LOGTAIL_TOKEN,
UCAN_LOG_BASIC_AUTH: env.UCAN_LOG_BASIC_AUTH,
UCAN_LOG_URL: env.UCAN_LOG_URL,

// These are injected in esbuild
// @ts-ignore
// eslint-disable-next-line no-undef
BRANCH: ACCOUNT_BRANCH,
BRANCH: env.ACCOUNT_BRANCH ?? '',
// @ts-ignore
// eslint-disable-next-line no-undef
VERSION: ACCOUNT_VERSION,
VERSION: env.ACCOUNT_VERSION ?? release.name ?? '',
// @ts-ignore
// eslint-disable-next-line no-undef
COMMITHASH: ACCOUNT_COMMITHASH,
COMMITHASH: env.ACCOUNT_COMMITHASH ?? release.gitRevShort ?? '',

PRIVATE_KEY: vars.PRIVATE_KEY,
DID: /** @type {UCAN.DID<"web">} */ (DID.parse(vars.DID).did()),
Expand Down
5 changes: 3 additions & 2 deletions packages/access-api/src/utils/context.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import {
} from '../models/delegations.js'
import { createD1Database } from './d1.js'
import { DbProvisions } from '../models/provisions.js'
import * as release from './release.build.js'

/**
* Obtains a route context object.
Expand Down Expand Up @@ -56,14 +57,14 @@ export function getContext(request, env, ctx) {
dsn: config.SENTRY_DSN,
debug: false,
environment: config.ENV,
release: config.VERSION,
release: release.name,
})

// Logging
const log = new Logging(request, ctx, {
token: config.LOGTAIL_TOKEN,
debug: config.DEBUG,
sentry: ['test', 'dev'].includes(config.ENV) ? undefined : sentry,
sentry: ['test'].includes(config.ENV) ? undefined : sentry,
branch: config.BRANCH,
version: config.VERSION,
commit: config.COMMITHASH,
Expand Down
9 changes: 9 additions & 0 deletions packages/access-api/src/utils/release.build.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
/**
* @file
* This file MAY be rewritten during build
* to make some buildtime variables available at runtime.
*/
/** @type {string|undefined} */
export const gitRevShort = undefined
/** @type {string|undefined} */
export const name = undefined
39 changes: 39 additions & 0 deletions packages/access-api/src/utils/release.node.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
/**
* @file
* utils related to sentry that rely on node.
* These might be used at build time,
* but won't work at run-time in cloudflare workers
*/
/* eslint-disable no-console */
/* eslint-disable unicorn/prefer-module */

// @ts-ignore
import git from 'git-rev-sync'
import path from 'path'
import { fileURLToPath } from 'url'
import { createRequire } from 'node:module'

const packageJson = createRequire(import.meta.url)('../../package.json')
const __dirname = () => path.dirname(fileURLToPath(import.meta.url))

/**
* Create a string to be used for the sentry release value
*
* @param {string} [env] - environment name e.g. 'dev'
* @param {object} pkg - package.json info
* @param {string} pkg.name
* @param {string} pkg.version
* @param {string} gitShort - git-rev-parse short value
* @returns {string} release name e.g. `@web3-storage__access-api@6.0.0-staging+92a89d3`
*/
export function getReleaseName(
env,
pkg = packageJson,
gitShort = git.short(__dirname())
) {
const version = `${pkg.name}@${pkg.version}-${env}+${gitShort}`.replace(
'/',
'__'
)
return version
}
23 changes: 12 additions & 11 deletions packages/access-api/wrangler.toml
Original file line number Diff line number Diff line change
@@ -1,15 +1,12 @@
# Development
name = "w3access-local"
account_id = "fffa4b4363a7e5250af8357087263b3a"
main = "./dist/worker.js"
main = "./src/index.js"

# Compatibility flags https://github.com/cloudflare/wrangler/pull/2009
compatibility_date = "2022-09-28"
compatibility_flags = ["url_standard"]

# We need to let wrangler bundle while D1 is in Beta
no_bundle = false

[[kv_namespaces]]
binding = "SPACES"
id = "e9fad7e04b254bf49206e08e50074387"
Expand All @@ -28,15 +25,16 @@ database_id = "7c676e0c-b9e7-4711-97c8-7b1c8eb229ae"
[[r2_buckets]]
binding = "DELEGATIONS_BUCKET"
bucket_name = "w3up-delegations-dev-0"
preview_bucket_name = "w3up-delegations-dev-0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes in this file seem unrelated to bundling / getting stack traces to work. not a blocker, but can you confirm why these changes are needed in this PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one was suggested by wrangler cli when I was deploying to dev env. The dev env has been usused for a bit, so I think we just hadn't exercised that path, so this makes sense as an addition that makes deploying to dev better (seemingly unrelated to d1/sentry/bundling, but the relationship is that the dev env is useful for testing deployments/new-code without disrupting staging, so it was useful here, and is probably useful to commit so others dont run into it when they try to use dev env + wrangler)


[vars]
ENV = "dev"
DEBUG = "true"
DID = "did:web:local.web3.storage"
PRIVATE_KEY="MgCYWjE6vp0cn3amPan2xPO+f6EZ3I+KwuN1w2vx57vpJ9O0Bn4ci4jn8itwc121ujm7lDHkCW24LuKfZwIdmsifVysY="
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same value from repo /.env.tpl , so it is already public

UPLOAD_API_URL = "https://up.web3.storage"

[build]
command = "scripts/cli.js build"
watch_dir = "src"

[miniflare]
Expand All @@ -47,24 +45,28 @@ d1_persist = ".wrangler/miniflare"
name = "w3access-dev"
workers_dev = true
vars = { ENV = "dev", DEBUG = "false", DID = "did:web:dev.web3.storage", UPLOAD_API_URL = "https://staging.up.web3.storage" }
build = { command = "scripts/cli.js build --env dev", watch_dir = "src" }
kv_namespaces = [
{ binding = "SPACES", id = "5697e95e1aaa436788e6d697fd3350be" },
{ binding = "VALIDATIONS", id = "ea17f472b37a43d29c1faf7af9512e03" },
]
d1_databases = [
{ binding = "__D1_BETA__", database_name = "access-dev", database_id = "4145a261-e54c-411d-a001-050fc30e4678" },
]
unsafe = { bindings = [
{ type = "analytics_engine", dataset = "W3ACCESS_METRICS", name = "W3ACCESS_METRICS" },
] }

[[env.dev.d1_databases]]
binding = "__D1_BETA__"
database_name = "access-dev"
database_id = "7f5c4ec7-610b-4885-b9f7-0886ce0639f6"

[[env.dev.r2_buckets]]
binding = "DELEGATIONS_BUCKET"
bucket_name = "w3up-delegations-dev-0"
preview_bucket_name = "w3up-delegations-dev-0"

# Staging
[env.staging]
name = "w3access-staging"
workers_dev = true
build = { command = "scripts/cli.js build --env staging", watch_dir = "src" }
kv_namespaces = [
{ binding = "SPACES", id = "b0e5ca990dda4e3784a1741dfa28a52e" },
{ binding = "VALIDATIONS", id = "b13f07c88fe848db9ccf651a0fea3fb6" },
Expand All @@ -90,7 +92,6 @@ bucket_name = "w3up-delegations-staging-0"
[env.production]
name = "w3access"
routes = [{ pattern = "access.web3.storage", custom_domain = true }]
build = { command = "scripts/cli.js build --env production", watch_dir = "src" }
kv_namespaces = [
{ binding = "SPACES", id = "5437954e8cfd4f7d98557132b0a2e93f" },
{ binding = "VALIDATIONS", id = "fb7cf10c725f45948321e88b8cb168ad" },
Expand Down