How to crawl a JavaScript website (React, Vue or Angular SPA) using Playwright

Classic crawlers such as wget --mirror, Scrapy or a fetch plus HTML parser download the HTML the server sends and look for <a href> tags in it. A server-rendered site puts its content and links in that HTML. A client-rendered single-page app sends something like this instead:

$ curl -s https://app.example.com/
<!doctype html>
<html>
  <head><script type="module" src="/assets/index-4f2a.js"></script></head>
  <body><div id="root"></div></body>
</html>

The navigation, the page content and every link only exist after index-4f2a.js runs, calls the API and renders. So the crawler reports one page and stops. To crawl a website built with JavaScript you need something that executes that script, which in practice means a real browser engine driven from code.

If you can change the app, server-side rendering or pre-rendering fixes this for every crawler at once. When you can't, or you need to crawl a JavaScript website as a logged-in user, use a headless browser.

This crawler does a breadth-first walk over same-origin links. It loads each URL in headless Chromium, waits for the network to go quiet, reads the rendered links and queues the new ones. It also records every fetch and XHR call it sees, which is often the most useful output. Save it as crawl.mjs and run node crawl.mjs https://your-site/.

import { chromium } from "playwright"
import robotsParser from "robots-parser"

const start = new URL(process.argv[2])
const MAX_PAGES = 50
const HASH_ROUTES = false // true for apps with URLs like /#/settings
const SKIP = /logout|sign-?out|delete|remove/i

const robotsUrl = new URL("/robots.txt", start).href
const robotsTxt = await fetch(robotsUrl).then((r) => (r.ok ? r.text() : ""), () => "")
const robots = robotsParser(robotsUrl, robotsTxt)
const DELAY_MS = Math.max(1000, (robots.getCrawlDelay("*") ?? 0) * 1000)

const key = (href) => {
  const url = new URL(href)
  if (!(HASH_ROUTES && url.hash.startsWith("#/"))) url.hash = ""
  return url.href
}

const browser = await chromium.launch()
const page = await browser.newPage()
const apiCalls = new Set()
page.on("request", (req) => {
  if (["fetch", "xhr"].includes(req.resourceType())) apiCalls.add(req.method() + " " + req.url())
})

const queue = [key(start.href)]
const seen = new Set(queue)
const pages = []

while (queue.length > 0 && pages.length < MAX_PAGES) {
  const url = queue.shift()
  if (robots.isAllowed(url, "*") === false) continue
  try {
    await page.goto(url, { waitUntil: "domcontentloaded", timeout: 30_000 })
    // Wait for the app's own API calls to settle, but don't hang on polling.
    await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {})
  } catch (error) {
    console.error("failed " + url + ": " + error.message)
    continue
  }

  const hrefs = await page.$$eval("a[href]", (links) => links.map((a) => a.href))
  pages.push({ url, finalUrl: page.url(), title: await page.title(), links: hrefs.length })

  for (const href of hrefs) {
    if (!href.startsWith("http")) continue
    const next = key(href)
    if (new URL(next).origin !== start.origin || SKIP.test(next) || seen.has(next)) continue
    seen.add(next)
    queue.push(next)
  }
  await page.waitForTimeout(DELAY_MS)
}

await browser.close()
console.log(JSON.stringify({ pages, apiCalls: [...apiCalls] }, null, 2))

A few choices in there are deliberate:

• networkidle waits until there have been no network connections for 500 ms. The Playwright docs discourage it for tests, but for a crawler it is a fair signal that the app has fetched its data and rendered. The 10 second timeout stops it hanging on apps that poll or hold a connection open.
• a.href inside $$eval returns the absolute URL the browser resolved, so relative links like ../settings need no extra handling.
• page.url() after loading shows where the app actually ended up. A client-side redirect to /login there means the crawler is not signed in.
• The SKIP pattern keeps the crawler away from logout and delete links, which matters as soon as you crawl a signed-in app.
• robots-parser reads Disallow and Crawl-delay. The robots.txt standard (RFC 9309) is short and worth a read if you crawl sites you don't own.

Puppeteer works the same way. Use page.goto(url, { waitUntil: "networkidle2" }) and page.$$eval with the same selector.

Most SPAs use the History API, so each screen has a real path such as /settings/billing. Loading that URL directly with page.goto makes the server return the app shell and the router renders the right screen. That is why the crawler above can treat SPA routes like ordinary pages.

Older apps and some dashboards use hash routing instead, with URLs like /#/settings. Everything after # never reaches the server, and most crawlers strip it, so every route collapses into one URL. Set HASH_ROUTES = true to keep fragments that start with #/ as separate pages. Playwright handles a navigation that only changes the hash, and the router renders the new route.

If a route only works after you click through from the home page (deep links return a 404 or a blank screen), the server is not configured to serve the app shell for every path. Crawl from the home page and click instead, as described in the next section.

• **Links that are buttons.** Some apps navigate with onClick={() => router.push("/reports")} on a <button> or <div>, with no href. A link-based crawler never sees those routes. Options are to click each candidate element and record page.url() afterwards, to read /sitemap.xml, or to find the route list in the app's source or JavaScript bundle.
• **Infinite scroll.** Items load as you scroll, so the links on page two of a list don't exist on first render. Scroll in a loop, for example await page.mouse.wheel(0, 5000) followed by a short wait, until the page height stops growing or you hit a cap.
• **Login.** A signed-out crawler sees the login screen on every route. Log in once in a script, save the session with await context.storageState({ path: "auth.json" }), and start later crawls with browser.newContext({ storageState: "auth.json" }). Keep the logout skip in place, or the first logout link ends the session.
• **Network idle that never comes.** Analytics beacons, WebSockets and polling can keep the network busy. The timeout in the example covers this. Waiting for a specific element with page.waitForSelector("main a") is more precise if you know the app.
• **URL explosions.** Filters, sort orders and tracking parameters can turn one list into thousands of URLs. Strip utm_ and similar parameters in key(), and keep the page cap.

A JavaScript app renders its pages from JSON it fetches. If what you want is the data, or the list of things the app can show, that JSON is usually cleaner and more stable than the rendered DOM. Class names and markup change with every redesign, while GET /api/projects?page=2 tends to stay put.

The apiCalls set in the crawler above is a starting point. Open the Network panel, click Fetch/XHR and use the app to see which endpoints feed each screen. How to find the API endpoints of a web app walks through that, and generate an OpenAPI spec from your traffic turns the calls into documentation. If you need the list of pages rather than the API, see how to find all pages on a website.

Netraptor is a Chrome extension with a crawler built for mapping the API a web app calls. It runs in your real Chrome tab, so React, Vue and Angular pages render as they normally do, with your logged-in session. There is no headless browser to install or sign in.

• Turn on Keep history for this site in the toolbar popup, then click Crawl this site, or open the Crawl tab in the Netraptor DevTools panel for more settings.
• It reads the a[href] links on the start page after that page has loaded and rendered in your tab. It keeps links on the exact same origin (subdomains are left out), or only those under the start page's path if you pick the Under a path prefix scope.
• It drops # fragments, tracking parameters such as utm_ and gclid, file downloads and links that look destructive, such as logout, delete, cancel and unsubscribe.
• Each queued link opens as a normal page load in your tab. Netraptor waits for Chrome to report the page loaded, then a fixed Wait per page that you set between 500 ms and 10 seconds.
• It records the fetch and XHR calls each page makes into a crawl session. The Crawl tab lists each visited page with its state, API call count and form count, and the API Map tab groups the calls into endpoints such as /orders/{id}.
• It notes the forms on each page. Filling and submitting them with test data is off by default, needs your confirmation for that origin, and never touches password, card, OTP, checkout or destructive-looking forms.

Know its limits. It visits the links found on the start page and does not follow links on the pages it visits, so start a second crawl from a section page to go further. It gives you the API calls per page, not an exportable list of page URLs. It doesn't click buttons or scroll, so routes behind onClick and items behind infinite scroll need a visit by hand, which Netraptor records too. Hash-routed apps collapse to one URL because fragments are dropped.

The Free plan crawls 25 pages per run. Pro ($29.99 one-time) crawls 100 pages by default and up to 1000 per run and adds OpenAPI 3.1, Bruno and Insomnia exports. Use it on apps you build, test or are authorised to test.