Scraping an Ecommerce Store with Browser Forest

The best first target for an ecommerce scraper is not a real marketplace — it's books.toscrape.com, a fake bookshop that the scraping community built to be scraped. It has a real storefront structure — category pages, product listings, star ratings, pagination, product detail pages — with none of the anti-bot walls, rate limiting or legal gray area of production stores. This tutorial runs against that site, so every snippet below is copy-paste runnable and fully legal.

You'll scrape the catalog three ways: list the first page, walk all 50 pages, and open a single product detail page. Same patterns you'd use on any store — minus the part where you get blocked.

What you need

• A Browser Forest API key — get one free (no card required)
• Python 3.9+, plus requests and Playwright:

pip install requests playwright
playwright install chromium

Step 1 — Connect and read the first page of books

POST /v1/sessions boots a patched Chromium instance in the cloud and blocks until it's live; connect_over_cdp() attaches Playwright over a real CDP connection — no injected shims, so the site's JavaScript sees a normal browser. The page itself is driven by Playwright, exactly as it would be against a local browser.

import requests
from playwright.sync_api import sync_playwright

session = requests.post(
    "https://browserforest.com/api/v1/sessions",
    headers={"X-API-Key": "bf_live_xxx"},
    json={},
).json()

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(session["cdpUrl"])
    page = browser.contexts[0].pages[0]
    page.goto("https://books.toscrape.com/")
    page.wait_for_load_state("networkidle")

    books = page.locator("article.product_pod")
    print(f"{books.count()} books on page 1")

    for i in range(3):
        book = books.nth(i)
        title = book.locator("h3 a").get_attribute("title")
        price = book.locator("p.price_color").inner_text()
        rating = book.locator(".star-rating").get_attribute("class")
        print(f"{title} — {price} — {rating}")

That's the whole scrape. Each book on the page is an article.product_pod card; the title lives in the h3 a link's title attribute, the price in p.price_color, and the rating is encoded as a class on .star-rating. Real stores nest this data differently, but the approach — find a repeating container, pull fields from it — is identical.

Step 2 — Walk all 50 pages

books.toscrape.com paginates 20 books per page across 50 pages. The next button lives in li.next a; when it stops existing, you're on the last page.

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(session["cdpUrl"])
    page = browser.contexts[0].pages[0]
    page.goto("https://books.toscrape.com/")
    page.wait_for_load_state("networkidle")

    seen = 0
    while True:
        for book in page.locator("article.product_pod").all():
            title = book.locator("h3 a").get_attribute("title")
            price = book.locator("p.price_color").inner_text()
            seen += 1
        nxt = page.locator("li.next a")
        if nxt.count() == 0:
            break
        nxt.click()                      # Playwright clicks the real <a>
        page.wait_for_load_state("networkidle")

    print(f"scraped {seen} books across {page.url}")

One detail worth stealing for production scrapes: we click the actual <a> element instead of constructing URLs. The browser resolves the relative href for us, which keeps the crawl correct even if the site changes its URL scheme.

Step 3 — Open a product detail page

Listings give you the cheap fields; the long description lives on each book's detail page. Click through and grab it:

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(session["cdpUrl"])
    page = browser.contexts[0].pages[0]
    page.goto("https://books.toscrape.com/")
    page.wait_for_load_state("networkidle")

    page.locator("article.product_pod h3 a").first.click()
    page.wait_for_load_state("networkidle")

    print("title:", page.locator("h1").inner_text())
    print("price:", page.locator(".price_color").inner_text())
    rating = page.locator(".star-rating").get_attribute("class")
    print("rating:", rating)
    desc = page.locator("#product_description + p").inner_text()
    print("description:", desc[:120], "...")

Going further: persistence and scale

Two Browser Forest features make the jump from tutorial to production painless:

• Persistent contexts. Some stores (and some pages of even practice-friendly sites) need a login cookie before they show everything. A POST /v1/contexts call gives you a durable cookie jar: attach its id to a session, log in once, then attach the same context to any number of later sessions — the login survives, so you never re-authenticate per run.
• Parallel sessions. Each POST /v1/sessions is an independent browser. To scrape 50 pages fast, shard the page range across a few sessions sharing one context — they don't share IP or fingerprint, and the worker keeps them isolated.

A note on scraping responsibly

We deliberately chose books.toscrape.com because its operator built it for exactly this purpose — norobots.txt restrictions, no terms-of-service ambiguity. That's the bar to hold any target to:

• Prefer sites that explicitly allow testing or offer an official API or sandbox.
• Check robots.txt and the terms of service before you automate anything.
• Never try to bypass a login or paywall — an authorized account with permission is a different thing from circumventing access controls.
• Throttle, identify yourself, and stop when asked. Good tooling makes being a good citizen easy: sessions auto-stop on idle_timeout, and DELETE /v1/contexts/<id> wipes saved state when you're done.

When a practice target is this close — real storefront structure, legal to scrape, and zero setup — there is no reason to cut your teeth on someone's production site. Run the snippets above, get your pipeline working end to end, and only then point it at targets you have permission for. The docs cover the rest of the API.