-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC 149 - Switch to per-page asset loading #152
Changes from all commits
fc1eec3
64480aa
35f516e
436828b
09a5bd2
03afdaa
07022a4
d751b79
b97a09a
adbc60a
779531b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,198 @@ | ||
--- | ||
Author: Ian James | ||
Date: 2 August 2022 | ||
Deadline for decision: 16 August 2022 | ||
--- | ||
|
||
# Serve component CSS and JavaScript as individual files | ||
|
||
## Summary | ||
|
||
Component stylesheets and JavaScript should only be included for components present on a page on GOV.UK. This will lead to a reduction in page size, improve the ability of the browser to cache assets - which in turn improves performance for user journeys and return visits - and help reduce our reliance on manual asset auditing. | ||
|
||
This RFC builds upon ideas and implementations from: | ||
* [RFC #91: Sharing assets][sharing_assets] (not implemented) | ||
* [RFC #108: Include specific component assets in applications][specific_assets] (implemented) | ||
* [RFC #115 : Enabling HTTP/2 on GOV.UK][http2] (implemented) | ||
|
||
## Problems | ||
|
||
### User journeys that visit more than one rendering app are inefficient | ||
|
||
Four out of ten tracked visitors to GOV.UK in the past year have visited more than one page in a single session. | ||
|
||
Each of the eleven rendering applications have different sets of CSS and JavaScript files containing both code specific to that application and code coming from shared components. | ||
|
||
These four out of ten visitors are likely to have visited pages rendered by multiple applications - which means they have downloaded the same code more than once via different files. | ||
|
||
For example, a three step journey can visit three different rendering applications: | ||
|
||
1. Homepage, rendered by [frontend] | ||
1. Search for "micro pig", rendered by [finder-frontend] | ||
1. Micro-pig guidance page, rendered by [government-frontend] | ||
|
||
Whilst there is shared code coming from [Static][static], CSS and JavaScript can still come from each individual application - which means there is overlap between the three applications' assets. In the previous three step journey example, there are 43 component's assets served; of which: | ||
|
||
* 6 components are used in all 3 applications | ||
* 16 components are used in 2 applications | ||
* 19 components are only used in 1 application | ||
|
||
That means that the code from 22 components is downloaded more than necessary. | ||
|
||
The following graph shows the rendering application's CSS size and the breakdown of that CSS: | ||
|
||
![](rfc-149/rendering-application-css-breakdown.svg) | ||
|
||
Around 15kB comes from the Static stylesheet - which is served from the same URI regardless of the rendering application, so can be cached by the browser and used again on all pages on GOV.UK. | ||
|
||
The applications have bespoke CSS - this varies between half a kilobyte to 15kB. This CSS can't be shared, as it's for pages only rendered by that application. | ||
|
||
The apps also have CSS that comes from the components that they use - this varies from half a kilobyte to 16kB. This is served in the rendering applications stylesheet - so a visitor who goes from app to app can't cache and then use these assets, despite the code being the same. | ||
|
||
The current setup has each application place their assets in a different folder - `/assets/collections`, `/assets/finder-frontend` etc. This prevents the browser cache from being used effectively as `/assets/collection/accordion-09288...adb36.css` will be fetched even if `/assets/finder-frontend/accordion-09288...adb36.css` has already been downloaded. | ||
|
||
### A tiny change in one component can prevent the browser using the whole of a cached JavaScript and CSS file | ||
|
||
GOV.UK doesn't take advantage of the fact that it serves CSS and JavaScript files with a long expiry header. Because the JavaScript and CSS is concatenated and fingerprinted, a change to one component will mean the filename will change. | ||
|
||
For example, any change to the breadcrumb component would cause a change in the CSS file for six rendering apps. Any return visitors would need to download the entire stylesheet again - serving the files individually would mean that only the updated breadcrumb component stylesheet would need to be downloaded. | ||
|
||
The component gem is released regularly - over the past year it's been released on average once every four days, with most releases being between one and seven days apart. Two thirds of repeat visitors to GOV.UK return within seven days and one quarter of repeat visitors return between 8 to 30 days. This means it is highly likely that returning visitors need to download the asset files again, regardless of how large or small any changes are. | ||
|
||
### Auditing component use in applications is a manual process. | ||
|
||
Each application only bundles the CSS and JavaScript for the components used - [see RFC #108 for more information][specific_assets]. | ||
|
||
Adding a component's CSS and JavaScript to an application's stylesheet and JavaScript is a manual process - which means that it's easy to forget to remove assets when a component has stopped being used in an application. We have auditing tools to help manage this - but they are not part of our deployment pipeline and rely on developers remembering to use them. | ||
|
||
Supplying the assets that a component needs as part of the partial would mean that assets would only be included when the component is used - this would simplify the developer experience when adding or removing components from applications, and would also mean a better user experience since unused assets wouldn't be loaded. | ||
|
||
## Proposal | ||
|
||
Assets should be included individually on a per-component basis and included in the component partial. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are we going to have something special for apps that use static here and components that are shared? I assume that by default we’ll end up with static including components and then a frontend app doing the same one. It looks like currently this is avoided by somewhat delicately not including static components in the list ? (which presumably means we risk the occasional breakage if rendering app and static are on different versions of the gem) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There isn't an overlap between components used in static and the rendering apps - static only does the header and the footer, so only renders (off the top of my head) the skip-link, navigation-header, footer, banners, and feedback components. These can use individual assets and won't clash with the rendering applications as the rendering applications don't use these components. We manually stick add components that are shared across a lot of the applications to stop the same CSS and JavaScript from being downloaded unnecessarily - this won't be needed in the future, so would stop any mismatched versions causing any breakages. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So components I'm referencing are things like input, hint, label. Those that are used as part of the components static uses and are then also used in applications. It's unclear to me how a rendering app would know that any static components used those components (well outside of hacking away at Slimmer) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm, not sure exactly how this would be done. We could:
I'd prefer the first option as it has no reliance on Slimmer - but any other suggestions welcome. The other thing worth noting is that this stops becoming a problem when Static/Slimmer is not longer being used - though that's probably a while off! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yup, I think the first option is preferable. Using slimmer makes me uncomfortable as it's very difficult to test and will likely be fragile.
I dream of the day! |
||
|
||
This would mean that a page loads `component.css` and `component.js` only when a component is present on the page - rather than the current behaviour, which sees component assets loaded if used anywhere within an application. If multiple uses of the same component occur on the page, there should only be one instance of the assets included on the page. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have some wonderings about dependencies here. Do we know: a) what CSS/JS is expected to be global, such as polyfills, govuk-frontend core, module system? i.e. things a component can use without requiring it? Would these become separate files to application.jss/css in apps or would there be extra files and/or ordering considerations ? We currently have the SCSS files of govuk_frontend_support and component_support that both output CSS and load in mixins. I’m guessing we’ll need something that does these tasks separately so apps can link to global dependencies while components have access to mixins? I worry that if all CSS components need a common pile of imports then this will create a big performance issue (with sprockets/libsass at least) - however if the dependencies can be light this shouldn’t be a problem, but that could be annoying for mixin availability. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Dependencies is a tricky one; JavaScript is a bit of an unknown as there are other factors outside of the RFC at play. a) For JavaScript, I'm not completely sure. What I hope is that with the potential to drop JavaScript support for Internet Explorer 11 we can investigate the use of ES6 modules. This means that including polyfills and dependencies becomes a lot easier: For CSS, we'd include everything needed from GOVUK Frontend - apart from components - into one file and serve that; this should include all the base styles ( b) There's an agreement that we don't use the Sass The common mixins would be included if needed by The lack of c) Yes, an
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, that sounds reasonable sounds like there's a fair amount of unknown. With JS, would we launch the individual files when we've got a similar set-up for JS and CSS. I.e individual files for each. Or would this propose doing just CSS and then leaving JS to later (so application.js entries for JS but individual files for CSS). I'm kinda hoping it'd be the former. One of the key places of all_components is in the component-guide itself, otherwise I think it's in backend apps that haven't had much love recently (Content Publisher, Content Data etc) If this work brings us closer towards Dart and/or modernising the JS that'd be a good outcome There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We'd do this for CSS first, and then tackle the JavaScript afterwards - so, as you say, there'd be Whilst it's not perfect, even just serving the CSS individually would be an improvement - and I worry that doing both at the same time would be too big a chunk of work to get done in a sensible time frame. Waiting for clearer direction for IE11 support, the upcoming translation work from GOVUK Frontend, and investigation into how the latest version of Rails uses ES6 modules would mean a longer wait until the JavaScript is loaded individually, but potentially less work to get it done. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ahh sure, that does seem somewhat messy as we'd have a separate install step for the JS and stylesheet. It seems ok to go with that as a first step and agree with your point about sensible time frame, but I really wouldn't wait for ES6 modules as we've got so many things to resolve to get to that. I imagine we could just create an equivalent JS file of component support and then just add each component as extra JS, similar to CSS. As an aside, I do wonder how we can start work on what JS without IE support looks like and the problems to overcome. |
||
|
||
### Shared assets | ||
|
||
The assets should be placed in a folder that allows each application to reference them. This feature is due as part of the replatforming work - but that should not block the serving of individual assets as there are performance advantages that don't rely on having a shared folder. | ||
|
||
A version identifier should be added to the component asset filename to help identify which version of the components gem is being used - for example `gem-v29-2-1-accordion-09288...adb36.css`. The fingerprint should remain to show that the contents of the file are the same. | ||
|
||
The bespoke app assets should be renamed to their application name - so collections would have `collections-5f801d...179b0.css` and `collections-828a8...abb6c.js`. This will make it easier to see which stylesheet comes from which application. | ||
|
||
Because each application is using the components gem as the source for the assets for each component, the compiled assets will be the same - whether the resulting CSS or JavaScript file is produced by whitehall, frontend, or collections. When writing to the shared folder the component assets may be overwritten, but - thanks to the fingerprint - they'll be overwritten with files containing the same content. | ||
|
||
## Considerations | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There probably should be something on impact to developer tooling as a consideration. I am a bit concerned about the impact we'll have on asset compilation times with the libsass / sprockets combination. We had to do a lot of work in the past to get these issues under wrap as they lead to failed test runs, apps that don't seem to run on first request etc once you end up with asset compilations drifting from seconds to minutes. There's a clearly a balance between application performance and ability to work with the application. So I'm wondering do we know yet enough about what the impact on development experience will be and, if not, if we're able to establish what would be an unacceptable regression? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think that this should be an optional choice and backwards compatible - so if that means running a A quick check of frontend running using app-live in GOVUK Docker, cold start:
Whilst not great, I think it's an acceptable increase whilst there are other avenues worth investigating to make this faster - off the top of my head, more efficient ways of only importing the specific things that are needed from GOVUK Frontend, using the faster Dart Sass, use of This is only a single run, so I'll do some more investigation into this tomorrow to be able to report an average number of runs. Whilst not a complete solution, running it using I'd suggest that an unacceptable regression would be if a cold start is above a minute under the current libsass and Docker set up. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Great. Yup having files without needing the import sounds great to avoid performance penalties everywhere. If it's a 25% increase as per your experiment that isn't too bad either - it's an order of magnitude better than I feared. A minute sounds good too, so long as we have a certain point where we're prepared to go "eep maybe we should focus on getting dart in first" works for me Yeah to do a full cold start you need to clear the cache, |
||
|
||
### Reduced compressibility when serving individual files | ||
|
||
GOV.UK uses [Brotli compression][enable_brotli] when serving assets - this is a compression method that works really well on repetitive text like HTML, CSS, and JavaScript. Serving individual files will reduce the amount of compression available because the files will have less repetition in them. Since each page will only be serving the component assets required on the page - rather than the components used in the entire application - there should be a reduction in asset size. | ||
|
||
To determine whether this is correct, I looked at the difference in CSS size for the top 100 most visited pages on GOV.UK over the past year (1 July 2021 to 31 June 2022). There was reduction in CSS size for pages[^1]. | ||
|
||
The largest change in component CSS size is -12.2kB, the smallest is -1kB, and the average decrease is 5.8kB; the graph shows the top 100 pages and the reduction in the amount of CSS for each page: | ||
|
||
![](rfc-149/css-size-for-top-100-most-visited-pages-on-govuk.svg) | ||
|
||
|
||
This means that all visitors will see a smaller page size, regardless of whether they visit one page or multiple pages. Six out of ten tracked visitors to GOV.UK only view a single page. | ||
|
||
Concatenating the assets can provide a benefit for those that visit multiple pages - but this doesn't work on GOV.UK due to the number of rendering apps used and the current lack of a shared space to put assets. | ||
|
||
### HTTP requests | ||
|
||
The number of requests per page would go up - this shouldn't be a problem as [GOV.UK has HTTP/2 enabled][http2], and is due to have [HTTP/3][http3] enabled in the near future. During the past month 99.5% of tracked visitors used browsers that support HTTP/2, and 66.2% of visitors used browsers that supported HTTP/3. | ||
|
||
Of the top 100 most visited pages on GOV.UK, the average number of components on a page is 19. The smallest number of components on a page is 9, and the largest number of components is 25. It's worth noting that most - but not all - components need a stylesheet (72 out of 78 components have a stylesheet) and not every component needs JavaScript (24 out of 78 components have a JavaScript file) - so adding a component to a page doesn't always mean two more requests are made. | ||
|
||
It's worth noting that this change will lead to slower load times for browsers that only support HTTP/1.1 - even when considering the reduction in page size. ([More information on HTTP/1.1's limitations][http_limitations]). This will impact Internet Explorer 11 on Windows 8 and below; the first versions of Chrome, Firefox, and Opera that supported HTTP/2 were released in early 2015. | ||
|
||
This balance between making the site load faster for a majority of users is not just a numbers game - the users who are most likely to be using older browsers are those that are likely to least afford the upgrade cost for a newer device, and they're likely to be the users who most need the information on GOV.UK. | ||
|
||
Whilst this proposed change will make things slower for users whose browser doesn't support HTTP/2 or HTTP/3, it won't prevent pages from loading. This proposed change should be accepted because it will dramatically improve page performance for almost all visitors without blocking access for a small number of visitors. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we have any estimates of how dramatic the improvement will be? I'd have thought this would only be of significance on slow connections/devices but would love to hear otherwise. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've included how much less CSS will be downloaded on the top 100 pages - all of the pages showed a reduction in the amount of CSS serve. But since the other part of this is to improve the cacheability of the files I don't think we'll be able to tell that until we've look at the real user metrics. A bit of a cop out I'm afraid! If you're on a fast-ish wired connection you probably won't notice any difference. If you're on a mobile connection it should really help, especially if it's flaky with a risk of both dropped packets and connections. |
||
|
||
### Non-rendering applications that use the components gem | ||
|
||
There are applications that use the components gem but don't use Static and Slimmer - for example, publishing applications and the Govspeak gem. | ||
|
||
These changes should be optional - even if this is the default behaviour, it should be able to be turned off at a component by component level and at an application level. This could be done with a helper method that checks both if the component has already been used, and if adding the component assets are allowed: | ||
injms marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```ruby | ||
# component_asset_helper.rb - pseudo code example: | ||
|
||
def component_assets_allowed (component_name) | ||
# check: | ||
# - if the component has already had the assets added to the page | ||
# - if the component assets loading method is allowed at an application level | ||
# - if the individual component has used an option to turn individual assets on or off | ||
@components_in_use.include?(component_name) || ENV.component_asset_loading !== "bulk" || use_individual_assets === true | ||
ChrisBAshton marked this conversation as resolved.
Show resolved
Hide resolved
|
||
end | ||
``` | ||
|
||
## Other solutions that have been considered and rejected | ||
|
||
* Put all components used across GOV.UK in one stylesheet and one JavaScript file - this increases page weight and doesn't solve cache invalidation | ||
* Serve one stylesheet and one JavaScript file for all of GOV.UK - this increases page weight and doesn't solve cache invalidation | ||
* Serve one stylesheet and one JavaScript file per template - this doesn't solve cache invalidation when one component changes | ||
* Serve all component assets used in an application in individual files - this increases page weight for single page visitors, though often decreased it for people who visited more than one page | ||
injms marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* Placing the stylesheet and JavaScript in the `body` directly before the component markup - whilst this is [okay][link_stylesheet_is_body_okay] for both stylesheets and `script`s, it leads to too many unknowns in what is already a reasonable large change | ||
|
||
## Implementation process | ||
|
||
Serving assets individually doesn't rely on the shared asset folder proposed in RFC 91 - so the lack of a shared folder is not a blocker. | ||
|
||
* implement the individual serving of assets | ||
1. Stylesheets | ||
1. JavaScript, as this will require more investigation and relies on external factors | ||
* once a shared folder is available, make all applications load assets from the shared folder | ||
|
||
### Implement the individual serving of assets | ||
|
||
Splitting this into two stages makes it more manageable and will allow improvements to be in stages. | ||
|
||
The stages themselves can be split into app-by-app upgrades - once an app is upgraded the pages it renders will see performance improvements without relying on any other app also being upgraded. This is a similar approach to the work that was done to use the public layout component - for example, [in collections][gem_layout_in_collections] and [in whitehall][gem_layout_in_whitehall] - so we know this strategy can work even if it takes time. | ||
|
||
This shouldn't be done on a component by component basis - changing only one component on a page will likely cause an increase in page size as the individual component assets can't take advantage of compression. | ||
|
||
There are more unknowns with the JavaScript implementation - and this aspect of the work may require further investigation and changes to applications. | ||
|
||
It's worth noting that ES6 modules and import maps are the [direction of travel that Ruby on Rails is taking][rails_direction], so this could be the path of least resistance that this implementation could take. GOV.UK Frontend also offers module based imports now, and has plans to drop JavaScript support for Internet Explorer 11 in the near future. It'd be worth seeing how close to a decision these are before choosing a direction. | ||
|
||
### Make assets load from a shared folder | ||
|
||
A shared folder will be implemented as part of the current replatforming work - so each application will be able to [upload any assets to a shared location][shared_location_diagram]. | ||
|
||
Once the shared folder is available, each application would need to load all their assets from a shared folder to allow the browser to cache assets regardless of rendering application. | ||
|
||
## Future plans | ||
|
||
With more confidence in the cachability of the individual component assets, there are further improvements to the performance of GOV.UK that could be made - for example: | ||
|
||
* preload assets based on most common next pages - similar to [guess.js][guessjs] | ||
* lazy loading of component assets that aren't visible - for example, the footer and feedback components are always at the bottom of a page | ||
|
||
|
||
[enable_brotli]: https://github.com/alphagov/govuk-rfcs/blob/6c16f831530be76a34954f30670035fcf7ae8ac1/rfc-138-enable-brotli-compression.md#L5-L6 | ||
[finder-frontend]: https://docs.publishing.service.gov.uk/repos/finder-frontend.html | ||
[frontend]: https://docs.publishing.service.gov.uk/repos/frontend.html | ||
[gem_layout_in_collections]: https://github.com/alphagov/collections/pull/2272 | ||
[gem_layout_in_whitehall]: https://github.com/alphagov/whitehall/pull/6240 | ||
[government-frontend]: https://docs.publishing.service.gov.uk/repos/government-frontend.html | ||
[govuk_publishing_components]: https://docs.publishing.service.gov.uk/repos/govuk_publishing_components.html | ||
[guessjs]: https://github.com/guess-js/guess | ||
[http_limitations]: https://github.com/alphagov/govuk-rfcs/blob/95b4f967a43b24141c4cd0c7feb37f3c309e21c8/rfc-139-enable-http3.md#http11 | ||
[http2]: https://github.com/alphagov/govuk-rfcs/blob/95b4f967a43b24141c4cd0c7feb37f3c309e21c8/rfc-115-enabling-http2-on-govuk.md | ||
[http3]: https://github.com/alphagov/govuk-rfcs/blob/95b4f967a43b24141c4cd0c7feb37f3c309e21c8/rfc-139-enable-http3.md | ||
[link_stylesheet_is_body_okay]: https://html.spec.whatwg.org/multipage/links.html#link-type-stylesheet | ||
[prerelease]: https://guides.rubygems.org/patterns/#prerelease-gems | ||
[rails_direction]: https://world.hey.com/dhh/modern-web-apps-without-javascript-bundling-or-transpiling-a20f2755 | ||
[shared_location_diagram]: https://docs.google.com/drawings/d/1UjW5El9AnrqzgdXjAXKfYBNFR0xNth54hT1PvnMsPxc/edit | ||
[sharing_assets]: https://github.com/alphagov/govuk-rfcs/blob/95b4f967a43b24141c4cd0c7feb37f3c309e21c8/rfc-091-sharing-assets.md | ||
[slimmer_moves_scripts]: https://github.com/alphagov/slimmer/blob/0968d5b715f949cc3ef5ac3fa1dcbababd7c2fd7/lib/slimmer/processors/tag_mover.rb#L8 | ||
[slimmer]: https://github.com/alphagov/slimmer/ | ||
[specific_assets]: https://github.com/alphagov/govuk-rfcs/blob/95b4f967a43b24141c4cd0c7feb37f3c309e21c8/rfc-108-including-gem-component-assets.md | ||
[static]: https://docs.publishing.service.gov.uk/repos/static.html | ||
[what_slimmer_does]: https://github.com/alphagov/slimmer/blob/0968d5b715f949cc3ef5ac3fa1dcbababd7c2fd7/docs/what-slimmer-does.md#tagmover | ||
|
||
[^1]: Method - look at the markup present on the page. Search for all instances of `.gem-c-*` to get a list of all components being used and dedupe. Compile the Sass from the GOV.UK Publishing Components gem into individual CSS files for each component. Get the Brotli-compressed file size for each component, and add it to the application-specific CSS. Compare this to the current CSS file size for the application and Static. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a big win 👏