Incremental Build RFC

[natemoo-re]

Summary

Built-in incremental build support

Links

• Rendered RFC
• Stage 2 Proposal
• Stage 1 Proposal

[montsamu]

One small datapoint in favor of the currently experimental content cache:

Audiobookaneers.com build:

21:10:55 [build] 5904 page(s) built in 170.55s

21:13:36 [build] 5904 page(s) built in 118.22s

Second build was more than 50 seconds (about 30%) faster. I'm hoping to see similar improvements on the Cloudfront side now that they have a "beta" build cache as well, but so far the gains are a bit more minimal (from 2m24s to 1m54s, though this DID include changes to 3 posts content, though not new posts/tags/etc) and their beta build cache only seems to be holding onto node_modules/.astro anyway, so far.

In the "best case scenario" when I add a post to /src/content/posts, the following pages would need to be rebuilt:

1. /2024/01/11/some-title/index.html -- the actual new post page
2. /2024/01/10/previous-title/index.html -- the previous new post needs rebuilt to pick up a "next newer post" link
3. /2024/01/11/ index.html -- index of the day's posts
4. /2024/01/11/page/2... -- in unlikely event of more than a page (10) worth of posts for the day, all page/2 page/3 etc. would need to be updated
5. /2024/01/index.html -- index of the month's posts
6. /2024/01/page/2... -- as above, the new post would bump all pages of this paginated set
7. /2024/index.html -- index of the year's posts
8. /2024/page/2... -- as above
9. /index.html -- front page, always shows newest posts
10. /page/2/index.html, /page/3/index.html... on through /page/70/index.html as the new post bumps through the pagination
11. /rss.xml
12. /author/myname/index.html -- archive by author
13. /author/myname/page/2...
14. /category/somecategory/index.html -- archive by category
15. /category/somecategory/page/2...
16. /tag/sometag/index.html -- archive by tag(s)
17. /tag/sometag/page/2...
18. /tag/anothertag...

I tell you what, this has discouraged me from implementing a 'recent posts' sidebar widget into the static build, as that would mean that every single page on the site would need to be regenerated. I'll definitely just use client-side JS for that! And it makes me think about investigating the "hybrid" approach for the pagination as this would cut down a lot of what needs to be rebuilt.

[Mark-RD]

Build caching idea:

//Return a page hash, unique for each page variation
export function getHash() {
  //return CMS.pageVersion;
  //return CMS.lastUpdateTime;
  //return null; //Always rebuild - Sitemap etc
  return Astro.props; //Default
}

//OR

export async function getStaticPaths() {
  return [
    { hash : /* required for partial building, null by default */ ,params: { /* required */ } ,props: { /* optional */ } },
  ];
}
---
TEMPLATE

If every page had a function (similar to getStaticPaths) called getHash that produced a unique hash for a particular page, then on a new build a list of the previous hashes and new hashes could be compared and only the ones with different hashes would be built.

This assumes that the output HTML is deterministic based of its inputs, for example the default implementation of getHash would return: sha256(Astro.props).

• Where to get the last array of page hashes? Build caches native to platforms
• Where to download the last full build to add the incremental files? Build caches native to platforms, or adaptors to S3 compatible APIs e.g. Cloudflare R2
• What if there are no build cache adaptors? Rebuild the full site
• What if the build cache is empty or expired? Rebuild the full site
• What if the hash array is empty? Rebuild the full site

[Mark-RD]

• What to do if the SSG is so large it times out current static host builds? Allow the build cache to be hydrated on a local machine before pushing to host

[jcayzac]

Suggested change

The original [incremental build support](https://github.com/withastro/roadmap/discussions/237) proposal is one of our oldest and most highly upvoted issues. The corresponding [roadmap issue](https://github.com/withastro/roadmap/issues/698) has likewise recieved a lot of attention and positive feedback. From the extensive performance profiling we've done, we know that Rollup is the main bottleneck in Astro's build process.

The original [incremental build support](https://github.com/withastro/roadmap/discussions/237) proposal is one of our oldest and most highly upvoted issues. The corresponding [roadmap issue](https://github.com/withastro/roadmap/issues/698) has likewise received a lot of attention and positive feedback. From the extensive performance profiling we've done, we know that Rollup is the main bottleneck in Astro's build process.

[jcayzac]

Suggested change

	- Restructure our
	- Restructure our (something?)

[ferparmur]

Suggested change

	This propsal introduces minimal public API surface changes, but requires many internal implementation details to be updated to support caching in some way.
	This proposal introduces minimal public API surface changes, but requires many internal implementation details to be updated to support caching in some way.

[matthewp]

The experimental feature was removed in Astro 5. With the introduction of Content Layer the internals of how Content Collections have changed quite a lot, so how an incremental build could work needs to be rethought.

[dmitryshostak]

Incremental builds on demand would be such a game-changer for many large companies and organizations with tens of thousands of static pages.

[KL13NT]

Do you think its possible to achieve simple ISR using app.render in a custom adapter? Would someone be willing to provide guidance for implementing that?

[delucis]

Do you think its possible to achieve simple ISR using app.render in a custom adapter? Would someone be willing to provide guidance for implementing that?

Here are a couple of guides to achieving something like ISR even with the regular adapters:

• https://logsnag.com/blog/implementing-isr-in-astro
• https://developers.netlify.com/guides/how-to-do-advanced-caching-and-isr-with-astro/

[KL13NT]

Thanks for the guides, but that's not exactly what I meant. I've already gone over the articles and they either use cache headers with SSR or stale-while-revalidate, or some Netlify specific stuff.

What I wish to explore is to build a node adapter (extension? Idk) that would look at incoming requests, determine the requested page, and render it if it has been flagged as revalidate by my revalidate api route. Is this even possible using the app.render() or did I misunderstand the utility?

@delucis

[delucis]

Ah I see. Yeah, I guess that might work? It might also not be needed if you follow a model more like the first guide link I shared. That shows invalidating the cache based on headers, but adding an endpoint that invalidates the cache for a page instead would also work in that model (and might be a bit easier to implement than rebuilding the adapter?)

[KL13NT]

That works when you can have a CDN in front of your server, which I believe is enough for the majority of users at this stage of Astro. Thanks for the suggestions. Can't wait for Astro-level ISR tho! 👀