How I broke Gatsby JS conditional page build and learned to debug the tool chain

...
info One or more of your plugins have changed since the last time you ran Gatsby. As a precaution, we're deleting your site's cache to ensure there's no stale data.
success initialize cache - 0.011s
...
success Building production JavaScript and CSS bundles - 30.860s
success Rewriting compilation hashes - 0.005s
...
success run queries - 49.184s - 44/44 0.89/s
success Building static HTML for pages - 17.453s - 40/40 2.29/s
success Delete previous page data - 0.000s
success Generating image thumbnails - 166.557s - 316/316 1.90/s
...
...
success onPreInit - 0.020s
success initialize cache - 0.012s
...
success Building production JavaScript and CSS bundles - 4.623s
success run queries - 4.804s - 4/4 0.83/s
success Building static HTML for pages - 0.732s - 0/0 0.00/s
success Delete previous page data - 0.002s
...

Comparing the two outputs I found these problems:

  1. Cache invalidation by changed plugins
    info One or more of your plugins have changed since the last time you ran Gatsby. As a precaution, we're deleting your site's cache to ensure there's no stale data.
  2. Rewriting compilation hashes
  3. Regenerating Images
    success Generating image thumbnails

creating a simpler setup

  1. create a gatsby site based on the default starter template
  2. upgrade gatsby and the tool chain to the latest stable versions
npx gatsby new gatsby-starter
cd gatsby-starter
yarn upgrade --latest
$ gatsby build --log-pages
success open and validate gatsby-configs - 0.034s
success load plugins - 1.153s
success onPreInit - 0.020s
success initialize cache - 0.012s
success copy gatsby files - 0.122s
success onPreBootstrap - 0.018s
success createSchemaCustomization - 0.008s
success source and transform nodes - 0.140s
success building schema - 0.535s
success createPages - 0.002s
success createPagesStatefully - 0.082s
success onPreExtractQueries - 0.002s
success update schema - 0.051s
success extract queries from components - 0.340s
success write out requires - 0.007s
success write out redirect data - 0.002s
success Build manifest and related icons - 0.211s
success onPostBootstrap - 0.220s

info bootstrap finished - 8.052s

success Building production JavaScript and CSS bundles - 6.059s
success Rewriting compilation hashes - 0.061s
success run queries - 6.757s - 7/7 1.04/s
success Building static HTML for pages - 0.904s - 4/4 4.43/s
success Delete previous page data - 0.003s
success Generating image thumbnails - 7.766s - 6/6 0.77/s
success onPostBuild - 0.018s
info Done building in 15.908345021 sec
info Built pages:
Updated page: /404/
Updated page: /
Updated page: /page-2/
Updated page: /404.html
✨ Done in 16.24s.
$ gatsby build --log-pages
success open and validate gatsby-configs - 0.033s
success load plugins - 0.827s
success onPreInit - 0.018s
success initialize cache - 0.007s
success copy gatsby files - 0.109s
success onPreBootstrap - 0.009s
success createSchemaCustomization - 0.006s
success source and transform nodes - 0.067s
success building schema - 0.375s
success createPages - 0.002s
success createPagesStatefully - 0.139s
success onPreExtractQueries - 0.004s
success update schema - 0.046s
success extract queries from components - 0.286s
success write out requires - 0.016s
success write out redirect data - 0.004s
success Build manifest and related icons - 0.223s
success onPostBootstrap - 0.249s

info bootstrap finished - 6.451s

success Building production JavaScript and CSS bundles - 4.623s
success run queries - 4.804s - 4/4 0.83/s
success Building static HTML for pages - 0.732s - 0/0 0.00/s
success Delete previous page data - 0.002s
success onPostBuild - 0.002s
info Done building in 12.096156853 sec
✨ Done in 12.45s.
  • Rewriting compilation hashes
  • Building static HTML 0/0
  • Generating Images

simulating the build process of the build Pipeline

  1. create temporal directory
  2. clone the git repository to a temporal directory
  3. restore the build cache
  4. run the build command
  5. save the build cache
#!/bin/bash
set -x
BUILD_CACHE=/tmp/buildcache.tgz
TMPDIR=$(mktemp -d)
git clone https://github.com/aheissenberger/gatsby-cache-problem.git $TMPDIR
if [ -f "$BUILD_CACHE" ]; then
tar -xf "$BUILD_CACHE" --directory $TMPDIR
fi
pushd $TMPDIR
yarn install
GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true yarn build --log-pages || exit 1
if [ -f "$BUILD_CACHE" ]; then
rm "$BUILD_CACHE"
fi
tar czf "$BUILD_CACHE" .cache/ public/
popd
...
success Building static HTML for pages - 1.621s - 0/0 0.00/s
success Delete previous page data - 0.004s
success Generating image thumbnails - 14.491s - 6/6 0.41/s <<==LOOK HERE
success onPostBuild - 0.012s
info Done building in 21.23928199 sec
✨ Done in 21.56s.
+ '[' -f /tmp/buildcache.tgz ']'
+ rm /tmp/buildcache.tgz
+ tar czf /tmp/buildcache.tgz .cache/ public/
+ popd

How to debug Gatsby JS in VS Code

{
"name": "Gatsby build Conditional",
"type": "node",
"request": "launch",
"protocol": "inspector",
"program": "${workspaceRoot}/node_modules/gatsby/dist/bin/gatsby",
"args": ["build", "--write-to-file", "--log-pages"],
"env": {"GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES":"true","CI":"true"},
"skipFiles": [],
"stopOnEntry": false,
"runtimeArgs": ["--nolazy"],
"sourceMaps": false
}

activate searching in the node_modules folder

{
"search.exclude": {
"**/node_modules":false
},
"search.useIgnoreFiles":false
}

Dig into the problem

  1. let’s search for Generating image thumbnails in the node_modules folder the result will list CHANGELOG.md and node_modules/gatsby-plugin-sharp/utils.js:38 looking into the second file will show us the code which prints the success message
  2. set a break point in this line [38]
  3. remove the .cache and public folder: rm -fr .cache/ ./public/
  4. start the VS Code Debugger and choose Gatsby build Conditional from the pulldown
  5. the debugger stops at line 38
  1. look at the CALL STACK in the left column and click on the second line - this is the script which called this function
actions.createJobV2 = (job, plugin) => (dispatch, getState) => {
const currentState = getState();
const internalJob = createInternalJob(job, plugin);
const jobContentDigest = internalJob.contentDigest; // Check if we already ran this job before, if yes we return the result
// We have an inflight (in progress) queue inside the jobs manager to make sure
// we don't waste resources twice during the process
if (currentState.jobsV2 && currentState.jobsV2.complete.has(jobContentDigest)) {
return Promise.resolve(currentState.jobsV2.complete.get(jobContentDigest).result);
}
// removed...
}
  • line 2 gets the state from the redux store which was populated from the .cache folder
  • line 3 creates a new Job object
  • line 4 gets the content digest hash from the new job
  • line 8 does a lookup in the redux store if the job has been done before
  1. set a breakpoint on const internalJob = createInternalJob(job, plugin);
  2. remove the .cache and public folder: rm -fr .cache/ ./public/
  3. run debugger — choose Gatsby build Conditional from the pulldown
  4. when the debugger pauses we can inspect the job attributes
internalJob.contentDigest = createContentDigest({
name: job.name,
inputPaths: internalJob.inputPaths.map(inputPath => inputPath.contentDigest),
outputDir: internalJob.outputDir,
args: internalJob.args,
plugin: internalJob.plugin
})

We found the Problem

Quick Fix

Long term Solutions to the Problem

  • convert the object to a JSON string and replace the path to the root folder with a stable string or hash (still only a quick fix)
  • do not allow to store any absolute path in any Gatsby data structures: Job objects, plugin objects

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Andreas Heissenberger

Andreas Heissenberger

Fast-track professional successful in the design, development and deployment of technology strategies and policy. Experienced leading Internet and IS operations