Incremental Build RFC

[natemoo-re]

Summary

Built-in incremental build support

Links

• Rendered RFC
• Stage 2 Proposal
• Stage 1 Proposal

[montsamu]

One small datapoint in favor of the currently experimental content cache:

Audiobookaneers.com build:

21:10:55 [build] 5904 page(s) built in 170.55s

21:13:36 [build] 5904 page(s) built in 118.22s

Second build was more than 50 seconds (about 30%) faster. I'm hoping to see similar improvements on the Cloudfront side now that they have a "beta" build cache as well, but so far the gains are a bit more minimal (from 2m24s to 1m54s, though this DID include changes to 3 posts content, though not new posts/tags/etc) and their beta build cache only seems to be holding onto node_modules/.astro anyway, so far.

In the "best case scenario" when I add a post to /src/content/posts, the following pages would need to be rebuilt:

1. /2024/01/11/some-title/index.html -- the actual new post page
2. /2024/01/10/previous-title/index.html -- the previous new post needs rebuilt to pick up a "next newer post" link
3. /2024/01/11/ index.html -- index of the day's posts
4. /2024/01/11/page/2... -- in unlikely event of more than a page (10) worth of posts for the day, all page/2 page/3 etc. would need to be updated
5. /2024/01/index.html -- index of the month's posts
6. /2024/01/page/2... -- as above, the new post would bump all pages of this paginated set
7. /2024/index.html -- index of the year's posts
8. /2024/page/2... -- as above
9. /index.html -- front page, always shows newest posts
10. /page/2/index.html, /page/3/index.html... on through /page/70/index.html as the new post bumps through the pagination
11. /rss.xml
12. /author/myname/index.html -- archive by author
13. /author/myname/page/2...
14. /category/somecategory/index.html -- archive by category
15. /category/somecategory/page/2...
16. /tag/sometag/index.html -- archive by tag(s)
17. /tag/sometag/page/2...
18. /tag/anothertag...

I tell you what, this has discouraged me from implementing a 'recent posts' sidebar widget into the static build, as that would mean that every single page on the site would need to be regenerated. I'll definitely just use client-side JS for that! And it makes me think about investigating the "hybrid" approach for the pagination as this would cut down a lot of what needs to be rebuilt.

[Mark-RD]

Build caching idea:

//Return a page hash, unique for each page variation
export function getHash() {
  //return CMS.pageVersion;
  //return CMS.lastUpdateTime;
  //return null; //Always rebuild - Sitemap etc
  return Astro.props; //Default
}

//OR

export async function getStaticPaths() {
  return [
    { hash : /* required for partial building, null by default */ ,params: { /* required */ } ,props: { /* optional */ } },
  ];
}
---
TEMPLATE

If every page had a function (similar to getStaticPaths) called getHash that produced a unique hash for a particular page, then on a new build a list of the previous hashes and new hashes could be compared and only the ones with different hashes would be built.

This assumes that the output HTML is deterministic based of its inputs, for example the default implementation of getHash would return: sha256(Astro.props).

• Where to get the last array of page hashes? Build caches native to platforms
• Where to download the last full build to add the incremental files? Build caches native to platforms, or adaptors to S3 compatible APIs e.g. Cloudflare R2
• What if there are no build cache adaptors? Rebuild the full site
• What if the build cache is empty or expired? Rebuild the full site
• What if the hash array is empty? Rebuild the full site

[Mark-RD]

• What to do if the SSG is so large it times out current static host builds? Allow the build cache to be hydrated on a local machine before pushing to host

[jcayzac]

Suggested change

The original [incremental build support](https://github.com/withastro/roadmap/discussions/237) proposal is one of our oldest and most highly upvoted issues. The corresponding [roadmap issue](https://github.com/withastro/roadmap/issues/698) has likewise recieved a lot of attention and positive feedback. From the extensive performance profiling we've done, we know that Rollup is the main bottleneck in Astro's build process.

The original [incremental build support](https://github.com/withastro/roadmap/discussions/237) proposal is one of our oldest and most highly upvoted issues. The corresponding [roadmap issue](https://github.com/withastro/roadmap/issues/698) has likewise received a lot of attention and positive feedback. From the extensive performance profiling we've done, we know that Rollup is the main bottleneck in Astro's build process.

[jcayzac]

Suggested change

	- Restructure our
	- Restructure our (something?)

[ferparmur]

Suggested change

	This propsal introduces minimal public API surface changes, but requires many internal implementation details to be updated to support caching in some way.
	This proposal introduces minimal public API surface changes, but requires many internal implementation details to be updated to support caching in some way.