Global data across the website

[jamesgorrie]

What's the problem

We need data across the site that is dynamic and generally fetched asyncronously.

These are

• Opening times in the footer from Prismic
• Global alert from Prismic
• Global popup from Prismic
• Toggles from here
• Prismic - an outlyer as it's not really data, but a class.

I see two approaches

• Cache data - find a way that's better than our current solution, storing data on the query param,
• Look at them individually - There are actually different solutions that might work

Cache data

Create another cache

There are quite a few options outlined in this discussion, before exploring any, I would like to get a taste for how people feel about solving these individually vs globally here and we should explore them.

Setting the requirements of this:

• data needs to be made available at build time
• data needs to be accessable to the client and server
• data needs to update outside of the build due to content being dynamic e.g. opening hours

1 & 2 - creating and accessing the data

An example of how to do this.

Something like next-plugin-preval is nice, but maybe adds overhead that we might not want.

There is the more more rudimentary webpack val-loader - usage explained here.

A nicer interface for something like ☝️ is hashicorp's prebuild-webpack-plugin.

The next step using any of the above is fiddly. How do we update the data incrementally?

Updating the data incrementally?

• Write and update a file cache on an interval

• Have an API endpoint that updates the data... this has many issues, one being you might only update the content one 1 instance / lambda etc, so ruling this out.

Incremental static regeneration

We could use . While this might work well for things like stories, and landing pages, and I think we should implement this here, it won't work well for things like search.

As this needs to be a global solution, this doesn't work.

---

Look at them individually

Opening times

???

Global alert

This could be async and built accessably.

Thoughts

• If someones network fails, they might miss the alert
• Jank
• It requires an async call on all pages. To help decrease the amount of data transferred to the browser we could have a separate api endpoint that returns 404 or similar if inactive.

Global popup

This could be async and built accessably.

Thoughts

• This is already JS only, and I think we are happy with this.
• Same as above, this would require an async call on all pages, but the amount of data transferred can be minimised by proxing a call that doesn't return anything if it's inactive.

Toggles

We could just bake the toggles into the app.

Thoughts

While the intention of the toggles livining in a different place was to be able to enable / disable them outside of deployment from a separate source, currently S3, this is never how we really do it.

The reason for this was the process of:

• develop feature behind a toggle
• turn toggle as default: on for everyone
• service fails
• change S3 object to default: off to stop broken service

It can also be useful for features that might degrade performance outside of our control, e.g. analytics code, sign ups, surveys, etc.

If we deem that this functionality is not required, we could bake them in.

Personally I think it's worth trying to finish this feature (it's has some holes in it) and improve it to make it more accessible, as I think that's the reason we don't use it.

Prismic

Currently we store a cached version of a JS object with methods you can call. This is an issue as it's not really data.

The issue however that we try to solve is that Prismic does two calls when looking up a document:

• getApi does an async call to get the current version, or ref, of the documents.
• getDocument (or similar GETs) with the ref fetched above to fetch that version of the document.

While I think caching of the object should not be included in this discussion, caching the ref could be.

[jamieparkinson]

Some clarifications would be helpful:

• Can you elaborate on "store the ref in the cache" re the prismic instance caching? imo caching that object is a different problem to data caching, that potentially has a different solution (because it can't be serialised and passed from server to client), and I might be inclined to remove it from the scope of this work.
• When you say "bake the toggles in", do you mean to set them at build time? Or have them set within the scope of the app? Not sure that I understand the problem with toggles as we currently have them.

---

I think removing dependency on runtime environment variables where possible and thus moving more towards static (incremental or otherwise) generation would be a good thing. Unfortunately this does mean that there would likely be a point where we'd need a different image for stage and prod, which (at least for me) we would need a very strong justification for.

I lean more towards wanting to solve this with caching etc, because it gives us the flexibility to add/change data without needing to think up new solutions. On the other hand, I don't think we should shy away from adding async calls to pages, so perhaps a combination of the approaches where we choose based on performance and network payload would work well.

[jamesgorrie]

@jamieparkinson I have added more description to the post opn Prismic, let me know if that's clear.

re: toggles - I mean don't have them exist in an external service.

Currently we:

1. lookup toggles at toggles.
2. check if a cookie matches any of those, override that value
3. send that data down on the query, like other cached data

By bake in - I mean override step 1. with

1. lookup cookies from module in codebase

As mentioned - there are reasons I would like to keep this service separate.

Going on what you're saying, toggles is something we need in the identity app and others, so I will try a few things trying to cache that.

We currently don't need opening times as we don't have a footer on those pages, which I think is wrong, but will confirm - but if we do, we will need a solution there too.

Sound sensible?

[jamieparkinson]

Yep understood re Prismic and toggles, thank you. It seems like we know we'll need some sort of global data cache (even if we're not sure what will be in it) so will be interesting to see what that might look like.

[jamesgorrie]

I've updated the discussion to put the cache challenge first, and started to address it.

I think the "Make the data" is easy. The "update the data" incrementally is harder.

[jamesgorrie]

@wellcomecollection/js-ts-reviewers I have an example repo here

This is mainly exploring the file system route as I couldn't get past even sharing the data with other methods.

The main restriction with the file system route is that it uses fs and path making it a little inaccessable to the client. This isn't an issue as

This means you can write server-side code directly in getServerSideProps. This includes reading from the filesystem or a database.

-- Source

We might be able to use those values from the data fetching methods and create a cache that is more accessible.

import fileStore from './fsStore'
import accessyCache from './accessyCache'

function getServerSideProps() {
  const toggles = fileStore.get('toggles')
  // this could use memory style cache as we're now in the context of the next app
  const accessyCache.set('toggles')
}

My suggestion would be that we implement the current fs cache. We could continue to use Contexts, but hoised to the the _app level, and continue to read them as we do.

If we wanted to remove those contexts - we could look at something similar to ☝️

[jamieparkinson]

Having also played around with it myself I can now see why the fs solution works now and the others don't. Still a little unclear on the exact proposal - are you saying that we should use _app-level Contexts that are populated by an fs-based cache? If so I'd agree; I've come full circle and now I reckon we should be using Contexts for global data, not aiming to get rid of them.

[jamieparkinson]

Would also add that any fs solution should definitely have some guards and helpful error messages regarding why it doesn't work on the client

[jamesgorrie]

Yup - that's mysuggestion, use FS, set the cache values within the server side data fetching methods, and populate the contexts from there.

The fs solution needs a better implementation, including better error handling, and probably memoization. The demo app was to explore what is available. I'll update it after though for future reference to others.

I'd like to make sure we check that this doesn't drop performance, as it does add I/O on disk, at least for initial render. Do we have a way of checking that pre-post to prod @jamieparkinson ?

[jamieparkinson]

Yep, APM (on the Node servers) will make that very clear. Tbh I'm kicking myself for only just adding it!