Build a technical SEO crawler that thinks like Googlebot

Commercial SEO tools are great at telling you something is wrong. I wanted a tool that tells the team exactly what Google would see, why it matters, and how to fix it — and that runs in CI against staging. So I built one: a Node.js crawl engine with a Next.js dashboard and a CLI on top.

One engine, two front ends

node bin/seo-audit.js https://example.com -p 300 -f html,json,csv

The CLI and the dashboard import the same engine. The dashboard adds tabs — Overview, Issues, Checklist, Pages, Crawl setup — and exports HTML, JSON or CSV. Keeping the engine UI-free made it testable and let the CLI run in pipelines.

robots.txt is harder than it looks

Most home-grown parsers get this wrong. Google's rules:

• Pick the most specific user-agent group that matches; groups don't merge with *.
• Within a group, the longest matching path wins; on a tie, Allow beats Disallow.
• Support * wildcards and the $ end anchor.
• Only the first 500 KiB is read.
• A 5xx on robots.txt means "assume everything is disallowed" for a while — a 404 means "allow everything".

function isAllowed(rules, path) {
  let best = { len: -1, allow: true };
  for (const r of rules) {
    if (!matches(r.pattern, path)) continue;
    const len = r.pattern.length;
    if (len > best.len || (len === best.len && r.allow)) best = { len, allow: r.allow };
  }
  return best.allow;
}

Checks people forget

• Soft 404s. Request a random URL like /this-page-should-not-exist-8f3a. If it returns 200, every "not found" page on the site is indexable junk.
• Host variants. http://, https://, www and bare domain should collapse to one with a single 301, not a chain.
• Canonical outside <head>. Google ignores it. So does a relative canonical in some setups.
• Content that needs JavaScript. Compare the raw HTML with the rendered DOM; if the main text only exists after JS, flag it.
• HTML over 2 MB. Googlebot stops reading there.
• CSS/JS blocked by robots.txt, which breaks rendering.

Every rule has a source

Each of the ~100 checks has a severity, a how-to-fix note and a link to Google Search Central, web.dev or the W3C spec. Checks based on convention instead of a documented Google rule are labelled (Heuristic) and capped at notice level. That one decision ended most "is this really a problem?" debates.

Core Web Vitals

Field data comes from the PageSpeed Insights API, with the standard thresholds: LCP ≤ 2.5 s, INP ≤ 200 ms, CLS ≤ 0.1. Lab checks — render-blocking scripts, images without dimensions, images over 200 KB, TTFB over 0.8 s — explain why a page fails.

The checks themselves are the machine-readable version of a technical SEO checklist for the App Router — building the crawler is what turns that list into something CI can fail on.

What I learned

1. Read the spec, not blog posts. Half the "SEO rules" online aren't Google rules.
2. Make severity honest. A tool that marks everything critical gets ignored.
3. Crawl like the bot: same user agent, same limits, same robots handling.