How I broke Gatsby JS conditional page build and learned to debug the tool chain

8 min readJun 1, 2020

I built my AWS CodeBuild Pipeline with a new feature called “ Conditional Page Builds” and it did not worked as expected in the build environment.

Starting the build process again with no change in the source code and with identical copies of the .cache and public folder generated this output:

...
info One or more of your plugins have changed since the last time you ran Gatsby. As a precaution, we're deleting your site's cache to ensure there's no stale data.
success initialize cache - 0.011s
...
success Building production JavaScript and CSS bundles - 30.860s
success Rewriting compilation hashes - 0.005s
...
success run queries - 49.184s - 44/44 0.89/s
success Building static HTML for pages - 17.453s - 40/40 2.29/s
success Delete previous page data - 0.000s
success Generating image thumbnails - 166.557s - 316/316 1.90/s
...

a second call to GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages immediately following the first one shows the expected output:

...
success onPreInit - 0.020s
success initialize cache - 0.012s
...
success Building production JavaScript and CSS bundles - 4.623s
success run queries - 4.804s - 4/4 0.83/s
success Building static HTML for pages - 0.732s - 0/0 0.00/s
success Delete previous page data - 0.002s
...

This proves the fact that the conditional page build is working and no page contains any data which is different on each creation.

Comparing the two outputs I found these problems:

Cache invalidation by changed plugins
info One or more of your plugins have changed since the last time you ran Gatsby. As a precaution, we're deleting your site's cache to ensure there's no stale data.
Rewriting compilation hashes
Regenerating Images
success Generating image thumbnails

To open an issue on github I had to create a simple setup simulating the problems with the environment.

Issue 1 and 2 (posted on github) where my fault by creating a temporal file with a name starting with gatsby-* which conflicted with a part of gatsby creating a hash on the content of all files starting with this name.

creating a simpler setup

create a gatsby site based on the default starter template
upgrade gatsby and the tool chain to the latest stable versions

npx gatsby new gatsby-starter
cd gatsby-starter
yarn upgrade --latest

on the first run of GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true yarn build --log-pages the result was as expected - there was no.cache and no public folder.

The build process creates the compilation hashes [24], the static HTML [26] and the image thumbnails [28].

$ gatsby build --log-pages
success open and validate gatsby-configs - 0.034s
success load plugins - 1.153s
success onPreInit - 0.020s
success initialize cache - 0.012s
success copy gatsby files - 0.122s
success onPreBootstrap - 0.018s
success createSchemaCustomization - 0.008s
success source and transform nodes - 0.140s
success building schema - 0.535s
success createPages - 0.002s
success createPagesStatefully - 0.082s
success onPreExtractQueries - 0.002s
success update schema - 0.051s
success extract queries from components - 0.340s
success write out requires - 0.007s
success write out redirect data - 0.002s
success Build manifest and related icons - 0.211s
success onPostBootstrap - 0.220s
⠀
info bootstrap finished - 8.052s
⠀
success Building production JavaScript and CSS bundles - 6.059s
success Rewriting compilation hashes - 0.061s
success run queries - 6.757s - 7/7 1.04/s
success Building static HTML for pages - 0.904s - 4/4 4.43/s
success Delete previous page data - 0.003s
success Generating image thumbnails - 7.766s - 6/6 0.77/s
success onPostBuild - 0.018s
info Done building in 15.908345021 sec
info Built pages:
Updated page: /404/
Updated page: /
Updated page: /page-2/
Updated page: /404.html
✨  Done in 16.24s.

running the same build command, a second time gives this output:

$ gatsby build --log-pages
success open and validate gatsby-configs - 0.033s
success load plugins - 0.827s
success onPreInit - 0.018s
success initialize cache - 0.007s
success copy gatsby files - 0.109s
success onPreBootstrap - 0.009s
success createSchemaCustomization - 0.006s
success source and transform nodes - 0.067s
success building schema - 0.375s
success createPages - 0.002s
success createPagesStatefully - 0.139s
success onPreExtractQueries - 0.004s
success update schema - 0.046s
success extract queries from components - 0.286s
success write out requires - 0.016s
success write out redirect data - 0.004s
success Build manifest and related icons - 0.223s
success onPostBootstrap - 0.249s
⠀
info bootstrap finished - 6.451s
⠀
success Building production JavaScript and CSS bundles - 4.623s
success run queries - 4.804s - 4/4 0.83/s
success Building static HTML for pages - 0.732s - 0/0 0.00/s
success Delete previous page data - 0.002s
success onPostBuild - 0.002s
info Done building in 12.096156853 sec
✨  Done in 12.45s.

as expected the following steps have been skipped:

Rewriting compilation hashes
Building static HTML 0/0
Generating Images

simulating the build process of the build Pipeline

create temporal directory
clone the git repository to a temporal directory
restore the build cache
run the build command
save the build cache

a simple bash script codepipeline.sh would look like this (don't use this for production as many edge cases are missing!):

#!/bin/bash
set -xBUILD_CACHE=/tmp/buildcache.tgz
TMPDIR=$(mktemp -d)
git clone https://github.com/aheissenberger/gatsby-cache-problem.git $TMPDIR
if [ -f "$BUILD_CACHE" ]; then
   tar -xf "$BUILD_CACHE" --directory $TMPDIR
fi
pushd $TMPDIR
yarn install
GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true yarn build --log-pages || exit 1
if [ -f "$BUILD_CACHE" ]; then
   rm "$BUILD_CACHE"
fi
tar czf "$BUILD_CACHE" .cache/ public/
popd

Run this script twice and you will see that on the second run the image generation still happens:

...
success Building static HTML for pages - 1.621s - 0/0 0.00/s
success Delete previous page data - 0.004s
success Generating image thumbnails - 14.491s - 6/6 0.41/s  <<==LOOK HERE
success onPostBuild - 0.012s
info Done building in 21.23928199 sec
✨  Done in 21.56s.
+ '[' -f /tmp/buildcache.tgz ']'
+ rm /tmp/buildcache.tgz
+ tar czf /tmp/buildcache.tgz .cache/ public/
+ popd

I documented my environment and posted an issue on github after not finding any other issue with a similar problem.

Since I am not very patient, I decided to look at the problem myself.

How to debug Gatsby JS in VS Code

The first thing I do when testing out a new framework is setting up the Javascript debugger in VS Code. Gatsby provides a very good documentation which makes this a breeze.

Add this to your existing VS Code launch.json to debug conditional page building:

{
   "name": "Gatsby build Conditional",
   "type": "node",
   "request": "launch",
   "protocol": "inspector",
   "program": "${workspaceRoot}/node_modules/gatsby/dist/bin/gatsby",
   "args": ["build", "--write-to-file", "--log-pages"],
   "env": {"GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES":"true","CI":"true"},
   "skipFiles": [],
   "stopOnEntry": false,
   "runtimeArgs": ["--nolazy"],
   "sourceMaps": false
}

activate searching in the `node_modules` folder

open the VS Code configuration .vscode/settings.json and add this:

{
   "search.exclude": {
      "**/node_modules":false
   },
   "search.useIgnoreFiles":false
}

Dig into the problem

The goal is to find the place in the code where the logic is which compares values from the cache with the existing environment and decides to rerun the image generation.

let’s search for Generating image thumbnails in the node_modules folder the result will list CHANGELOG.md and node_modules/gatsby-plugin-sharp/utils.js:38 looking into the second file will show us the code which prints the success message
set a break point in this line [38]
remove the .cache and public folder: rm -fr .cache/ ./public/
start the VS Code Debugger and choose Gatsby build Conditional from the pulldown
the debugger stops at line 38

look at the CALL STACK in the left column and click on the second line - this is the script which called this function

We see that this code part of the gatsby-plugin-sharp is activated by the event CREATE_JOB_V2

If this would have been a normal function call, simple looking up the CALL STACK would be easy, but this is an event which could have been raised anywhere in the code based

Cancel the debugger and search again for CREATE_JOB_V2 inside the node_modules folder the search result contains 6 files where node_modules/gatsby/dist/redux/actions/public.js looks most promising

looking at the 8 lines of this function:

actions.createJobV2 = (job, plugin) => (dispatch, getState) => {
   const currentState = getState();
   const internalJob = createInternalJob(job, plugin);
   const jobContentDigest = internalJob.contentDigest; // Check if we already ran this job before, if yes we return the result
   // We have an inflight (in progress) queue inside the jobs manager to make sure
   // we don't waste resources twice during the processif (currentState.jobsV2 && currentState.jobsV2.complete.has(jobContentDigest)) {
      return Promise.resolve(currentState.jobsV2.complete.get(jobContentDigest).result);
   }
   // removed...
}

line 2 gets the state from the redux store which was populated from the .cache folder
line 3 creates a new Job object
line 4 gets the content digest hash from the new job
line 8 does a lookup in the redux store if the job has been done before

set a breakpoint on const internalJob = createInternalJob(job, plugin);
remove the .cache and public folder: rm -fr .cache/ ./public/
run debugger — choose Gatsby build Conditional from the pulldown
when the debugger pauses we can inspect the job attributes

The left column shows us the following attributes with absolute paths /User/xxxxxxxx/...:
job.inputPaths, job.outputDir, plugin.pluginFilepath, plugin.resolve

Step into the function createInternalJob to check on the creation of the diggest

scrolldown until you see the place where the contentDigest is created:

internalJob.contentDigest = createContentDigest({
   name: job.name,
   inputPaths: internalJob.inputPaths.map(inputPath => inputPath.contentDigest),
   outputDir: internalJob.outputDir,
   args: internalJob.args,
   plugin: internalJob.plugin
})

We found the Problem

The hash created by Gatsby Job Queue Manager will always break when compared to values in the cache which have been created in a source folder with a different absolute path as long as the Job and the Plugin object contain absolute paths.

I also create an issue on github where you can follow the next steps.

Quick Fix

Always copy the source folder to the same path which has been used when the .cache folder was created.

Long term Solutions to the Problem

convert the object to a JSON string and replace the path to the root folder with a stable string or hash (still only a quick fix)
do not allow to store any absolute path in any Gatsby data structures: Job objects, plugin objects

I hope I could give some people a first impression on how to track down a problem by yourselves and provide maintainers of a project a better documentation to fix architectural problems.

Originally published at https://www.heissenberger.at.