How to scrape a website that requires a login (and why the API is usually easier)

A login usually means the data belongs to someone. Scrape it when it is your site, your own account's data, or the site's terms allow automated access. Check robots.txt (the format is in RFC 9309, and Python ships urllib.robotparser to read it), keep your request rate low and look for an official API or export before writing a scraper.

Use your own credentials and never share them or someone else's. Do not try to get around CAPTCHAs, paywalls or access controls. If a site puts a CAPTCHA in front of its login, take it as a no.

Most sites keep you logged in in one of three ways. Knowing which one you are dealing with decides how your script stays logged in.

• A session cookie. The login form POST returns Set-Cookie, and the browser sends that cookie with every later request. This is the classic server-rendered site, and requests.Session handles it.
• A CSRF token alongside the cookie. The login form carries a hidden input such as csrf_token, _token, authenticity_token or csrfmiddlewaretoken, and the server rejects a POST without it. Some apps put it in a <meta name="csrf-token"> tag and expect it back in an X-CSRF-Token header.
• A bearer token. Single-page apps often call a JSON login endpoint, keep the returned token in localStorage and send it as Authorization: Bearer ... on each API call. Cookies alone will not log you in here.

To see which one you have, log in with the Network panel open and Preserve log checked. The login request's Headers tab shows any Set-Cookie response headers, its Payload tab shows the fields the form sent, and the API calls after it show whether an Authorization header appears. The Chrome Network reference covers these tabs.

For a classic form login, requests.Session keeps cookies between requests, and BeautifulSoup reads the CSRF token out of the form. Install both with pip install requests beautifulsoup4. Copying every named input from the form picks up the CSRF token and any other hidden fields without knowing their names in advance.

import os
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

BASE = "https://app.example.com"

session = requests.Session()
session.headers["User-Agent"] = "orders-export/1.0 (you@example.com)"

# GET the login page first so the server sets its cookie and CSRF token
resp = session.get(f"{BASE}/login", timeout=30)
resp.raise_for_status()
form = BeautifulSoup(resp.text, "html.parser").find("form")

payload = {
    field["name"]: field.get("value", "")
    for field in form.find_all("input")
    if field.get("name")
}
# Use the field names you saw in the login request's Payload tab
payload["email"] = os.environ["APP_EMAIL"]
payload["password"] = os.environ["APP_PASSWORD"]

action = urljoin(resp.url, form.get("action") or resp.url)
login = session.post(action, data=payload, timeout=30)
login.raise_for_status()

# Many sites answer a failed login with 200 and the login page again
if "/login" in login.url:
    raise SystemExit("Login failed: check field names, CSRF token and credentials")

page = session.get(f"{BASE}/account/orders", timeout=30)
soup = BeautifulSoup(page.text, "html.parser")
for row in soup.select("table.orders tbody tr"):
    print([td.get_text(strip=True) for td in row.find_all("td")])

If the token lives in a meta tag instead, read it with soup.find("meta", attrs={"name": "csrf-token"})["content"] and send it as a header on the POST. The requests documentation on Session objects covers cookie persistence and default headers.

This approach only sees the HTML the server sends. If the page you want is empty in View Source and fills in after load, the data comes from a JavaScript API call. Use Method 3 for that, or Playwright if you really need the rendered page.

Logging in on every run is slow and can trip login rate limits or security alerts. Save the session's cookies after a successful login and load them next time, then log in again only when they stop working.

import json
from pathlib import Path

COOKIES = Path("cookies.json")

# after a successful login
COOKIES.write_text(json.dumps(session.cookies.get_dict()))

# next run
session = requests.Session()
session.cookies.update(json.loads(COOKIES.read_text()))

get_dict() drops each cookie's domain and path, which is fine for a script that talks to one site. You can also copy the Cookie request header from a logged-in request in DevTools and set it on the session, as a quick test before writing the login code.

When the login runs through JavaScript, or the data only appears after the page renders, drive a real browser. Playwright for Python can save a logged-in context to a JSON file and start new contexts from it. Install it with pip install playwright and then playwright install chromium.

import os
from pathlib import Path

from playwright.sync_api import sync_playwright

BASE = "https://app.example.com"
STATE = Path("state.json")

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)

    if not STATE.exists():
        context = browser.new_context()
        page = context.new_page()
        page.goto(f"{BASE}/login")
        page.get_by_label("Email").fill(os.environ["APP_EMAIL"])
        page.get_by_label("Password").fill(os.environ["APP_PASSWORD"])
        page.get_by_role("button", name="Sign in").click()
        page.wait_for_url("**/dashboard")
        context.storage_state(path=STATE)
        context.close()

    context = browser.new_context(storage_state=STATE)
    page = context.new_page()
    page.goto(f"{BASE}/account/orders")
    for text in page.locator("table.orders tbody tr").all_inner_texts():
        print(text)
    browser.close()

storage_state saves cookies and localStorage, so it also covers single-page apps that keep their token in localStorage. It does not save sessionStorage. If the app keeps its token there, you have to log in each run or read the token in the page and set it yourself.

Selenium works the same way. Log in, save driver.get_cookies() to a file, and on the next run open any page on the site and call driver.add_cookie() for each saved cookie before going to the page you want.

Two-factor codes, passkeys and single sign-on through Google, Okta or Microsoft are hard to automate and are not meant to be. The practical answer is to log in by hand once in the Playwright browser and save the state:

context = browser.new_context()
page = context.new_page()
page.goto(f"{BASE}/login")
input("Log in in the browser window (2FA, SSO and all), then press Enter here")
context.storage_state(path=STATE)

Your script then reuses state.json until the session expires, and you repeat the manual login when it does. For a CAPTCHA on the login page, do the same and solve it yourself once. Do not wire in a CAPTCHA-solving service.

Most modern apps render their pages from JSON API calls. Requesting that JSON gives you clean, structured data with field names, and it does not break when the page layout changes. To find the endpoint:

• Open DevTools on the logged-in page, go to Network and click Fetch/XHR.
• Reload the page, or click the button that loads the data you want.
• Click each request and check its Preview tab until you find the one holding the data on screen. Pressing Ctrl+F (Cmd+F on a Mac) in the Network panel searches response bodies too, so you can type a value you can see on the page.
• Note the URL, method, query parameters and any request headers that look app-specific, such as Authorization, X-CSRF-Token or X-Requested-With.

How to find the API endpoints of a website covers this search in more detail. Then call the endpoint with the session you already logged in with:

resp = session.get(
    f"{BASE}/api/orders",
    params={"page": 1, "per_page": 50},
    headers={"Accept": "application/json"},
    timeout=30,
)
resp.raise_for_status()
orders = resp.json()

If you logged in with Playwright, context.request.get(f"{BASE}/api/orders") sends the context's cookies for you, and .json() on the response gives you the data. To hand the Playwright login to requests instead, copy the cookies from the saved state:

state = json.loads(STATE.read_text())
for c in state["cookies"]:
    session.cookies.set(c["name"], c["value"], domain=c["domain"], path=c["path"])

For an app that sends a bearer token, cookies are not enough. Copy the Authorization header from the request in DevTools to test, then read the token from the login response or from localStorage in the saved state for the real script.

A shortcut while exploring: right-click the request in DevTools and choose Copy as cURL. Run it in a terminal to confirm the endpoint works outside the browser, then translate it to Python by hand or with a converter such as curlconverter. The copied command contains your cookies, so do not paste it anywhere public.

Sessions end. Your script will eventually get a 401, a 403 or a redirect back to the login page. Check for those on every response, log in again once and retry the request, and stop if the second attempt fails too.

Go slowly. Put a pause between requests, use the API's own page size instead of hammering it with parallel calls, and respect 429 Too Many Requests:

import time

items, page = [], 1
while True:
    r = session.get(f"{BASE}/api/orders", params={"page": page}, timeout=30)
    if r.status_code == 429:
        time.sleep(int(r.headers.get("Retry-After", "30")))  # assumes the seconds form
        continue
    r.raise_for_status()
    batch = r.json()["items"]
    if not batch:
        break
    items.extend(batch)
    page += 1
    time.sleep(1)

Set a descriptive User-Agent with a way to contact you. It costs nothing and gives the site owner a way to reach you before they block you.

A saved state.json or cookies.json is as good as your password until the session ends. Treat it that way.

• Read the email and password from environment variables or a secrets manager, as in the examples above.
• Add state.json, cookies.json and .env to .gitignore before the first commit.
• Strip Cookie and Authorization headers from anything you share, including cURL commands, HAR files and bug reports.
• Use a separate test account for anything you run on a schedule, so a leaked session does not expose your main account.

If the site is your own app or one you are authorised to test, the slow part of Method 3 is finding every endpoint and its parameters. Netraptor is a Chrome extension that records the fetch and XHR calls your logged-in app makes and groups them for you. Everything it records stays in your browser.

• Open the Netraptor panel in DevTools and it captures the tab's API calls live. Turn on Keep history for this site in the toolbar popup and it also records them across page loads with DevTools closed.
• The API Map tab groups calls into endpoints by method and path, with IDs replaced, such as GET /api/orders/{id}. Each endpoint shows the request and response schema inferred from real traffic, the actual responses it returned and the pages that called it.
• Right-click a request to copy it as cURL (bash or cmd), PowerShell, fetch or Node.js fetch. There is no Python option, so convert the cURL command as described above.
• The Export tab writes a Postman Collection v2.1, a HAR file or curl commands, with Strip auth headers and cookies on by default. OpenAPI 3.1 export is part of Pro.

To find endpoints on pages you would not think to open, run Crawl this site. It collects the links on the start page (one level deep, same origin only) and visits them in your own logged-in tab, so it uses your real session. To cover a deeper section, start the crawl from that section's page. It skips links that look destructive, such as logout, delete, cancel and unsubscribe. The Free plan crawls 25 pages per run, and Pro crawls 100 by default and up to 1000. Form filling is off until you allow it for a site in the Crawl tab, and even then it never submits forms with password, card or one-time-code fields, or checkout and destructive forms. The API calls from every visited page land in the API Map.

Related reading: find all pages on a website, crawl a JavaScript website and create a Postman collection from your browser.