How to find all pages on a website, including hidden and logged-in pages

No single source has every page. Each method below sees a different slice of the site, so you combine them and remove duplicates.

• Sitemap: the pages the site owner wants search engines to find. Fast and complete for content sites, often missing for apps.
• Google site: search: pages Google has indexed. A sample, never the full count.
• Search Console: the exact indexed and not-indexed list. Only for sites you own or have been added to.
• Crawler: every page reachable by following links. Misses pages nothing links to and links that only exist in JavaScript.
• Wayback Machine: URLs the Internet Archive has seen over the years, including deleted ones.
• Logged-in browser crawl: pages and API calls that only exist after you sign in. Only for your own or authorised apps.

This guide is for site owners, developers, SEO and QA teams mapping sites they run or are allowed to test. The same methods work on public sites you don't own, but stick to what's public and respect robots.txt.

Most CMSs and site builders publish a sitemap automatically. Try /sitemap.xml first, then /sitemap_index.xml. If neither loads, open /robots.txt. The Sitemap: line there gives the real address, and a site can list several.

A sitemap file holds up to 50,000 URLs and 50 MB uncompressed, according to the sitemaps.org protocol. Bigger sites split their URLs into many files and publish a sitemap index, a <sitemapindex> file whose <loc> entries point at the other sitemaps. Open each child sitemap to get the page URLs.

To pull the URLs out of a sitemap on the command line:

curl -s https://example.com/sitemap.xml \
  | grep -oE '<loc>[^<]+' \
  | sed 's/<loc>//' > urls.txt

A sitemap shows what the owner chose to list. Draft pages, account pages and old pages that were never removed from the server are usually missing, and stale sitemaps can list pages that now 404.

Type site:example.com into Google to see pages it has indexed for that domain. Add a path to focus on one section, such as site:example.com/docs, or combine it with a word that appears on the pages you want.

It's quick and works on any site, but the count is not the real number of pages. Google's own site: operator documentation says it "doesn't necessarily return all the URLs that are indexed" under the prefix you searched. Pages blocked from indexing, behind a login, or too new to be crawled won't show up at all.

If you own the site, Google Search Console gives a far better list than site:. Open Indexing, then Pages. The Page indexing report charts indexed and not-indexed pages and lists why each group of URLs isn't indexed, such as Not found (404), Excluded by noindex tag, or Crawled - currently not indexed.

Click a reason to see example URLs, then use Export. The example lists are capped at 1,000 rows each, so very large sites still need a crawl or the sitemap to fill the gaps. The not-indexed groups are often where forgotten pages turn up.

A crawler starts at one URL, collects every link on the page, visits each one and repeats until it runs out of new links. The result is every page reachable from the start page, which is usually close to the real site for content sites.

Screaming Frog SEO Spider is the common desktop choice. The free version crawls up to 500 URLs per crawl, and a paid licence removes the limit. Enter the home page, start the crawl and filter the Internal tab to HTML to get the page list, then export it.

For a free command-line crawl, wget can spider a site without saving the pages:

wget --spider --recursive --level=inf --no-verbose \
  --output-file=crawl.log https://example.com/

grep -oE 'https?://[^ ]+' crawl.log | sort -u > urls.txt

wget follows robots.txt rules when it crawls recursively, and it does not run JavaScript. Both crawlers find links in the HTML the server sends. If the site builds its menus in JavaScript, see how to crawl a JavaScript website.

The Internet Archive has been capturing pages for decades, and its CDX server API lists every URL it captured for a domain. Ending the URL with /* asks for a prefix match, fl=original returns only the URLs, and collapse=urlkey drops repeat captures of the same URL:

https://web.archive.org/cdx/search/cdx?url=example.com/*&fl=original&collapse=urlkey&filter=statuscode:200

This is the best way to find all URLs of a website that existed at some point, such as old campaign pages or a blog section that was removed from the menu. Expect a lot of noise from query strings and assets. Check which URLs still load before you treat them as live pages.

A "hidden" page is usually an orphan page: it exists on the server, but no menu, link or sitemap points to it. Crawlers can't find it because there is no link to follow. What you can do depends on whether the site is yours.

• On your own site, compare lists. URLs in your CMS, your server access logs or your analytics that never show up in a crawl are orphan pages.
• On any site, the Wayback Machine and site: search often reveal pages that were linked once and later unlinked.
• robots.txt sometimes names paths the owner asked search engines to skip. That tells you the path exists, but it is not an invitation to open it.

Guessing URLs with a wordlist (directory brute forcing) sends thousands of requests for paths that may not exist. Do it only on your own site or a target you are authorised to test, such as a scoped security engagement. On someone else's site it can break their terms and trip their security monitoring.

Two kinds of sites defeat every method above.

Single-page apps (React, Vue, Angular and similar) often define their routes in JavaScript. A link crawler that reads the server's HTML sees an empty shell with one <div> and a script tag. Routes reached through buttons with click handlers, rather than real <a href> links, never appear as links at all. Sitemaps for these apps tend to cover only the marketing pages.

Logged-in apps hide most of their pages behind a session. A crawler without your cookies gets redirected to the login page on every URL. You can sometimes paste a session cookie into a crawler, but sessions expire, and you still don't see which API calls each screen makes.

If you just need the links on one rendered page, open the Console in Chrome DevTools on that page and run this. It reads the live DOM, so links your app drew with JavaScript are included:

[...new Set([...document.querySelectorAll('a[href]')].map(a => a.href))].join('\n')

Chrome may ask you to type allow pasting first. Here is why. For the full scraping side of logged-in sites, see how to scrape a website behind a login.

Netraptor is a Chrome extension for developers that records the API calls a web app makes. Its crawler runs in your own tab, so it uses the login you already have, and it records what each page calls while it visits. It's built for apps you build, test or are authorised to test.

• Start it from the toolbar popup with Crawl this site, or from the Crawl tab in the Netraptor DevTools panel. The first time, Netraptor asks for access to that site.
• It reads the <a href> links on the start page after the page has rendered, so links your single-page app drew with JavaScript count. It keeps links on the same origin and drops # fragments, tracking parameters such as utm_ and gclid, and file downloads like PDFs and images.
• It skips links that look destructive: anything with logout, sign-out, delete, remove, destroy, cancel, unsubscribe or deactivate in the path or query.
• It loads each link in your tab, waits (500 to 10,000 ms per page, set in the Crawl tab) and records the calls. A badge in the page shows progress, and you can stop and resume.
• Form filling is off by default. Turn it on for a domain and confirm, and it submits forms with test data. It never submits forms with password, card or one-time-code fields, or forms that look like checkout or a delete action.

The Crawl tab lists every page it visited with its state (Done or Error), the number of API calls and the number of forms. Click a page to see its requests. The API Map then groups the calls from all pages into endpoints such as /orders/{id}, which you can export to Postman, HAR or curl. More on that in how to find the API endpoints of a web app.

Know the limits. The crawler visits the links found on the page you start from and doesn't follow links from those pages in turn, so start from a section index or pick Under a path prefix to cover one area. Routes that only exist as #/hash URLs or click handlers aren't followed. The Free plan crawls 25 pages per run and tells you how many links it found. Pro ($29.99 one-time) crawls up to 1000 pages per run.

If your coding agent needs the result, here is how to give it the recorded API calls.