After a deployment, the first visitor pays for the cold cache. This tool does that visit for them: it crawls a page, follows every link it finds, and tracks what it has already seen.
Configuration is environment variables — the entry page and the link selector — which made it easy to aim at different sites and stages.
| Fact | Detail |
|---|---|
| Behaviour | Crawls an entry page and all discovered links |
| Configuration | `CRAWL_PAGE`, `CRAWL_LINKSELECTOR` |
| Stack | Node (crawler, seenreq) |
| Started | March 2018 |
What's in it
The link selector decides what counts as a page to warm, so it adapts to different site structures.
Small enough to run as the last step of a release.
In detail
What is actually inside it
What counts as a page to warm is a CSS selector, so the same binary handles a blog, a shop and a documentation site without code changes.
Small enough to run as the last step of a release — the point was that the first visitor after a deploy should not pay for a cold cache.
| Detail | Value |
|---|---|
| Behaviour | crawl entry page + all matched links, deduplicated |
| Configuration | `CRAWL_PAGE`, `CRAWL_LINKSELECTOR` |
| Started | March 2018 |
Technical notes
The decisions behind it
Why warm caches at all?
Why deduplicate URLs?
seenreq, sites with tag pages and paginated archives turn into infinite loops. Deduplication is what makes it safe to point at a real site.Under the hood
crawl.js, read literally
The whole tool is crawl.js: 72 lines, two runtime dependencies (crawler, seenreq), no dev dependencies and no tests. It seeds the queue with CRAWL_PAGE + "/", so the configured value has to stay a bare origin — a trailing slash produces a // entry URL.
Each response is parsed with Cheerio and $(CRAWL_LINKSELECTOR) picks the elements whose href becomes a candidate. A candidate survives only if it clears eleven substring tests: it must not point at another origin and must not contain javascript:, mailto:, file:, tel:, fax:, ?, typo3/ or fileadmin/. The "other origin" test is not a URL comparison — it is linkUrl.indexOf(PAGE) > -1, i.e. "does the configured origin appear somewhere in this string?"
Deduplication is a bare seenreq instance: seen.exists(PAGE + relLinkUrl) checks and records in the same call, hashing a normalised request into an in-memory object. crawler's own skipDuplicates is left at its false default, so this one call is the only thing standing between the run and a loop.
The crawler options set exactly two things: HTTP Basic auth and rateLimit: 300. Everything else is the library default — maxConnections: 10, timeout: 15000, retries: 3, retryTimeout: 10000. The rate limit is what you actually notice: measured against a local server, requests leave 277–301 ms apart, so throughput is about 3.3 pages per second and the ten connections are never exercised. When the queue drains, the process calls exit(0).
What happens when a crawled URL is not HTML?
crawler skips Cheerio injection for non-injectable content types and invokes the callback with response.$ undefined, after logging response body is not HTML, skip injecting. The extraction block is guarded with typeof $ === "function", but the log line right after it is not: it calls $("title").text(). One link to a PDF, image or JSON endpoint therefore throws TypeError: $ is not a function inside the crawl callback and kills the process with exit code 1. Reproduced with a page linking to an application/pdf response — the run died after three URLs.What happens if the credentials are unset?
auth object is passed unconditionally, with user/pass read straight from CRAWL_USER/CRAWL_PASSWORD. With both undefined, request throws no auth mechanism defined for every URL. crawler retries three times, ten seconds apart, then logs the error; the queue drains and the process still exits 0. A misconfigured run spends ~30 seconds warming nothing and reports success to whatever invoked it.Why can it never warm a URL that uses a query string?
href containing ? before it is ever queued, so ?page=2, filter and sort links and TYPO3/Extbase plugin URLs are silently skipped. The fragment strip (#) and the leading-slash fix run only for links that got past this gate. The corollary is that the warmer can only ever reach URLs the static HTML already exposes.Does it execute JavaScript or respect robots.txt?
robots.txt fetch, and the requests carry no User-Agent at all (verified: only host, authorization, accept-encoding, connection).A deploy step in a TYPO3 image
what the ignore-list and the Dockerfile give away
typo3/ and fileadmin/, pin the target down: a TYPO3 site. typo3/ is the backend entry point and fileadmin/ the uploads directory, so skipping them kept the crawl out of the CMS internals and away from binary assets. The point of running it was to make the front-end cache hot the moment a release finished, so it ran as a one-shot at the end of a deploy, not as a service.FROM node:carbon, matching .nvmrc's lts/carbon (Node 8, long past EOL). npm install rather than npm ci; the --only=production line is left commented out and node_modules is excluded by .dockerignore. The image existed so the crawl could be the last step of a release without a Node runtime on the deploy host.
There is no CI config anywhere in the tree — build and trigger sat in the surrounding pipeline, and the Dockerfile was the contract: build it, run it once with CRAWL_PAGE, CRAWL_LINKSELECTOR and credentials.
The same credentials are attached to every request, so a staging site behind htpasswd warms cleanly. A cookie session or a form login does not — those pages simply never appear in the crawl, and nothing reports that they were missed.
README.md is a single heading; the operating knowledge was never written down. The eslint config targets ecmaVersion: 5 and the stylelint config loads stylelint-scss plugins for a repo that contains no stylesheets — leftovers from the project template it grew out of.
Links
Where to find this one