Price Scraping: How To Build a Scraper, Test It, and Scale With Confidence

TL;DR

• Price scraping is the automated extraction of product pricing data, including name, price, currency, availability, and discount status from eCommerce websites
• This guide builds a working Python scraper using Playwright for rendering and BeautifulSoup for parsing, with structured JSON output via Pydantic
• Real eCommerce sites use CAPTCHA, IP blocks, JavaScript rendering, and obfuscated selectors. Each obstacle has a concrete fix.
• Rotating residential proxies, randomized request timing, and realistic browser fingerprints are the three biggest factors in staying undetected
• AI cuts the manual work of selector generation, price normalization, and anomaly detection, but adds cost and latency, so use it selectively
• Price scraping legality varies by jurisdiction and target site ToS. Amazon explicitly prohibits it; publicly available product prices are generally lower risk.
• For production use, store selectors in config files, validate every output record, monitor for silent failures, and schedule scrapes based on data volatility
• For high-value or heavily protected targets, Decodo's eCommerce price scraper API returns structured pricing data without the maintenance burden

What is price scraping, and why does it matter

Price scraping is the process of collecting product pricing data from eCommerce websites using automated tools. A typical scrape gathers the product name, current price, currency, availability, and discount status. This data is collected repeatedly and at scale from hundreds or thousands of product pages.

Price scraping is a type of web scraping focused specifically on structured product data that updates often and has strong protection against bots. Unlike general scraping, which might collect blog content or contact details, price scraping targets information that impacts revenue decisions. As a result, the websites that try to protect this data take their defenses seriously.

Why businesses use it

Competitive price monitoring

Retailers keep an eye on their competitors' prices to make quick, informed decisions about their own pricing. For example, a UK electronics retailer tracks GPU prices at Currys and Argos every day. This allows them to match a competitor's price drop within hours instead of days. Being able to respond this quickly gives them a strong advantage in the market.

MAP enforcement

Brands use price scraping to ensure that authorized sellers follow minimum advertised price rules. Checking hundreds of reseller websites by hand isn't practical, so automated scraping helps identify violations quickly. This way, brands can address issues before they harm their reputation.

Deal alerts and price tracking

Services that help consumers send alerts when a product's price falls below a set level. The scraper runs on a schedule, compares the current price against a stored baseline, and fires an alert when the condition is met. This logic is simple, but it needs reliable and accurate data to work well.

Market research

Aggregating prices from many sellers shows trends that you can't see by just looking at a single product. This includes patterns like seasonal changes, price differences by region, and shifts across different product categories. For analysts and pricing teams, this information is essential for building a solid strategy.

If you're interested in scraping eCommerce websites, for more information beyond pricing, that guide offers a complete overview. And if you're running any kind of monitoring use case, proxies for real-time price monitoring cover the infrastructure side.

Building the price scraper step by step

This section explains how to build a price scraper in Python. We'll use Playwright to display the website and BeautifulSoup to extract information. Our target site is books.toscrape.com, which is a practice eCommerce site set up for scraping. You can develop without worrying about any terms of service issues.

Setting up the project

Create a dedicated project directory and set up a virtual environment before installing anything. This keeps your dependencies isolated from other Python projects on your machine.

mkdir price-scraper
cd price-scraper

mkdir price-scraper
cd price-scraper

Create and activate a virtual environment:

# Mac/Linux
python3 -m venv venv
source venv/bin/activate

# Windows (WSL)
python3 -m venv venv
source venv/bin/activate

# Mac/Linux
python3 -m venv venv
source venv/bin/activate

# Windows (WSL)
python3 -m venv venv
source venv/bin/activate

You should see (venv) appear at the start of your terminal prompt, which confirms the virtual environment is active. Any packages you install from this point are scoped to this project only.

Create the main script file:

touch price_scraper.py

touch price_scraper.py

Your project directory should now look like this:

price-scraper/
├── venv/
└── price_scraper.py

price-scraper/
├── venv/
└── price_scraper.py

Now install the dependencies:

pip install playwright beautifulsoup4 pydantic requests dotenv
python -m playwright install chromium

pip install playwright beautifulsoup4 pydantic requests dotenv
python -m playwright install chromium

The ppython -m playwright install chromium step downloads the necessary Chrome browser that Playwright uses. This browser is different from any other browsers on your computer, and the download takes about a minute to finish.

Setting up your proxy credentials

The scraper routes requests through Decodo's residential proxies to rotate IPs automatically and avoid blocks. Before you write any scraping code, get your credentials from the Decodo dashboard:

1. Log in to your Decodo dashboard.
2. Navigate to Residential → Proxy setup in the left sidebar.
3. Scroll to the Authentication section, your first proxy username and password are created automatically.
4. Copy your username and password. Click the eye icon to reveal the password, or click directly on it to copy.

Your endpoint and port are fixed. gate.decodo.com:7000 is the default residential proxy gateway for rotating requests. Only the username and password are specific to your account.

Before entering your credentials into the scraper, make sure your proxy is working. Go to the Proxy setup page, scroll to Code examples, choose Python, copy the snippet, and run it.

import requests

username = "your_username"
password = "your_password"
proxy = f"http://{username}:{password}@gate.decodo.com:7000"

result = requests.get("https://ip.decodo.com/json", proxies={
    "http": proxy,
    "https": proxy
})

print(result.text)

import requests

username = "your_username"
password = "your_password"
proxy = f"http://{username}:{password}@gate.decodo.com:7000"

result = requests.get("https://ip.decodo.com/json", proxies={
    "http": proxy,
    "https": proxy
})

print(result.text)

If the response shows a different IP address from your own, the proxy is working correctly.

Store your credentials in a .env file instead of hardcoding them into your script. This is especially important if you plan to share your code publicly. 

To do this, create a .env file in the root of your project:

DECODO_PROXY_USERNAME=your_username
DECODO_PROXY_PASSWORD=your_password

DECODO_PROXY_USERNAME=your_username
DECODO_PROXY_PASSWORD=your_password

Add .env to your .gitignore so it never gets committed:

echo ".env" >> .gitignore

echo ".env" >> .gitignore

Load them in your script:

import os
from dotenv import load_dotenv

load_dotenv()

proxy = {
    "server": "http://gate.decodo.com:7000",
    "username": os.getenv("DECODO_PROXY_USERNAME"),
    "password": os.getenv("DECODO_PROXY_PASSWORD")
}

import os
from dotenv import load_dotenv

load_dotenv()

proxy = {
    "server": "http://gate.decodo.com:7000",
    "username": os.getenv("DECODO_PROXY_USERNAME"),
    "password": os.getenv("DECODO_PROXY_PASSWORD")
}

Navigating to product pages and handling dynamic content

To start, run Playwright in headless mode. Headless mode means the browser operates in the background without a visible window. This makes it faster and better for automated scraping.

import asyncio
import os
from dotenv import load_dotenv
from playwright.async_api import async_playwright

load_dotenv()

async def launch_browser():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            proxy={
                "server": "http://gate.decodo.com:7000",
                "username": os.getenv("DECODO_PROXY_USERNAME"),
                "password": os.getenv("DECODO_PROXY_PASSWORD")
            },
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
        )
        page = await context.new_page()
        return browser, context, page

import asyncio
import os
from dotenv import load_dotenv
from playwright.async_api import async_playwright

load_dotenv()

async def launch_browser():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            proxy={
                "server": "http://gate.decodo.com:7000",
                "username": os.getenv("DECODO_PROXY_USERNAME"),
                "password": os.getenv("DECODO_PROXY_PASSWORD")
            },
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
        )
        page = await context.new_page()
        return browser, context, page

Block unnecessary resources to speed up page loads

Images, fonts, and stylesheets add load time without adding useful data. Block them:

async def block_resources(page):
    await page.route(
        "**/*",
        lambda route: route.abort()
        if route.request.resource_type in ["image", "font", "stylesheet"]
        else route.continue_()
    )

async def block_resources(page):
    await page.route(
        "**/*",
        lambda route: route.abort()
        if route.request.resource_type in ["image", "font", "stylesheet"]
        else route.continue_()
    )

Navigate to the product listing page and wait for prices to render:

import asyncio
import random

async def navigate_to_page(page, url):
    await page.goto(url, wait_until="domcontentloaded")
    
    # wait for price elements to render
    await page.wait_for_selector(".price_color", timeout=10000)
    
    # randomized delay to mimic human browsing patterns
    await asyncio.sleep(random.uniform(2, 5))
    
    return await page.content()

import asyncio
import random

async def navigate_to_page(page, url):
    await page.goto(url, wait_until="domcontentloaded")
    
    # wait for price elements to render
    await page.wait_for_selector(".price_color", timeout=10000)
    
    # randomized delay to mimic human browsing patterns
    await asyncio.sleep(random.uniform(2, 5))
    
    return await page.content()

The wait_for_selector function pauses the program until the price element shows up on the page. This is important for prices that load with JavaScript through AJAX, as it ensures the scraper only tries to read the page after the data has loaded. The random delay of 2 to 5 seconds simulates how a real person browses, making it harder for anti-bot systems to identify the scraping activity.

Parsing price data from the page

After capturing the page content, pass the source to Beautiful Soup to begin the extraction process. BeautifulSoup is a Python library that converts messy HTML into a navigable structure, allowing you to isolate specific product data using CSS selectors with precision.

from bs4 import BeautifulSoup

def parse_prices(html):
    soup = BeautifulSoup(html, "html.parser")
    products = []
    
    for article in soup.select("article.product_pod"):
        try:
            # extract product name
            name = article.select_one("h3 a")["title"]
            
            # extract and normalize price
            raw_price = article.select_one(".price_color").text.strip()
            price = float(raw_price.replace("£", "").replace(",", ""))
            
            # extract availability
            availability = article.select_one(".availability").text.strip()
            
            # extract rating
            rating = article.select_one("p.star-rating")["class"][1]
            
            products.append({
                "name": name,
                "price": price,
                "currency": "GBP",
                "availability": availability,
                "rating": rating
            })
            
        except (AttributeError, TypeError, ValueError):
            # skip products with missing or malformed data
            continue
    
    return products

from bs4 import BeautifulSoup

def parse_prices(html):
    soup = BeautifulSoup(html, "html.parser")
    products = []
    
    for article in soup.select("article.product_pod"):
        try:
            # extract product name
            name = article.select_one("h3 a")["title"]
            
            # extract and normalize price
            raw_price = article.select_one(".price_color").text.strip()
            price = float(raw_price.replace("£", "").replace(",", ""))
            
            # extract availability
            availability = article.select_one(".availability").text.strip()
            
            # extract rating
            rating = article.select_one("p.star-rating")["class"][1]
            
            products.append({
                "name": name,
                "price": price,
                "currency": "GBP",
                "availability": availability,
                "rating": rating
            })
            
        except (AttributeError, TypeError, ValueError):
            # skip products with missing or malformed data
            continue
    
    return products

A few things worth noting here:

• CSS selectors help target elements by their class name and structure. For example, the selector article.product_pod targets all product cards on the page. If you want to learn more about choosing the right selector, see XPath vs. CSS selectors.
• When it comes to prices, you should normalize them by removing the currency symbol and converting the string into a float. Different countries have different ways of writing numbers. For instance, 1.299,00 is the German format, while 1,299.00 is the US format. Always normalize the price before saving it.
• Using try/except is a good practice as it allows you to handle issues if an element is missing without crashing the entire scraping process. Learn more about data parsing and its importance.

Structuring and saving the output

Define a Pydantic model to ensure that every scraped record has a consistent structure. Pydantic is a Python library that validates data types while the program runs. For example, if a price comes back as a string instead of a float, it catches the error before it affects your output file.

from pydantic import BaseModel
from typing import Optional
from datetime import datetime

class ProductPrice(BaseModel):
    product_name: str
    price: float
    currency: str
    original_price: Optional[float] = None
    discount: Optional[float] = None
    availability: str
    source_url: str
    scraped_at: datetime = datetime.utcnow()

from pydantic import BaseModel
from typing import Optional
from datetime import datetime

class ProductPrice(BaseModel):
    product_name: str
    price: float
    currency: str
    original_price: Optional[float] = None
    discount: Optional[float] = None
    availability: str
    source_url: str
    scraped_at: datetime = datetime.utcnow()

Serialize the output to a timestamped JSON file:

import json
from datetime import datetime

def save_to_json(products, source_url):
    timestamp = datetime.utcnow().strftime("%Y%m%d_%H%M%S")
    filename = f"prices_{timestamp}.json"
    
    validated = []
    for product in products:
        try:
            record = ProductPrice(
                product_name=product["name"],
                price=product["price"],
                currency=product["currency"],
                availability=product["availability"],
                source_url=source_url,
            )
            validated.append(record.model_dump())
        except Exception as e:
            print(f"Validation error: {e}")
            continue
    
    with open(filename, "w") as f:
        json.dump(validated, f, indent=2, default=str)
    
    print(f"Saved {len(validated)} products to {filename}")
    return filename

import json
from datetime import datetime

def save_to_json(products, source_url):
    timestamp = datetime.utcnow().strftime("%Y%m%d_%H%M%S")
    filename = f"prices_{timestamp}.json"
    
    validated = []
    for product in products:
        try:
            record = ProductPrice(
                product_name=product["name"],
                price=product["price"],
                currency=product["currency"],
                availability=product["availability"],
                source_url=source_url,
            )
            validated.append(record.model_dump())
        except Exception as e:
            print(f"Validation error: {e}")
            continue
    
    with open(filename, "w") as f:
        json.dump(validated, f, indent=2, default=str)
    
    print(f"Saved {len(validated)} products to {filename}")
    return filename

Print a formatted summary to the console so you can sanity-check results without opening the file:

def print_summary(products):
    print(f"\n{'Product':<50} {'Price':>10} {'Availability':<20}")
    print("-" * 82)
    for p in products[:10]:  # show first 10
        print(f"{p['product_name'][:48]:<50} {p['currency']} {p['price']:>7.2f} {p['availability']:<20}")

def print_summary(products):
    print(f"\n{'Product':<50} {'Price':>10} {'Availability':<20}")
    print("-" * 82)
    for p in products[:10]:  # show first 10
        print(f"{p['product_name'][:48]:<50} {p['currency']} {p['price']:>7.2f} {p['availability']:<20}")

Full script put it all together:

import asyncio
import json
import os
import random
from datetime import datetime
from typing import Optional

from bs4 import BeautifulSoup
from dotenv import load_dotenv
from playwright.async_api import async_playwright
from pydantic import BaseModel

load_dotenv()


class ProductPrice(BaseModel):
    product_name: str
    price: float
    currency: str
    original_price: Optional[float] = None
    discount: Optional[float] = None
    availability: str
    source_url: str
    scraped_at: datetime = datetime.utcnow()


def parse_prices(html, source_url):
    soup = BeautifulSoup(html, "html.parser")
    products = []

    for article in soup.select("article.product_pod"):
        try:
            name = article.select_one("h3 a")["title"]
            raw_price = article.select_one(".price_color").text.strip()
            price = float(raw_price.replace("£", "").replace(",", ""))
            availability = article.select_one(".availability").text.strip()

            record = ProductPrice(
                product_name=name,
                price=price,
                currency="GBP",
                availability=availability,
                source_url=source_url,
            )
            products.append(record.model_dump())

        except (AttributeError, TypeError, ValueError):
            continue

    return products


def save_to_json(products):
    timestamp = datetime.utcnow().strftime("%Y%m%d_%H%M%S")
    filename = f"prices_{timestamp}.json"
    with open(filename, "w") as f:
        json.dump(products, f, indent=2, default=str)
    print(f"Saved {len(products)} products to {filename}")


def print_summary(products):
    print(f"\n{'Product':<50} {'Price':>10} {'Availability':<20}")
    print("-" * 82)
    for p in products[:10]:
        print(
            f"{p['product_name'][:48]:<50} "
            f"{p['currency']} {p['price']:>7.2f} "
            f"{p['availability']:<20}"
        )


async def main():
    url = "https://books.toscrape.com/catalogue/category/books_1/index.html"

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            proxy={
                "server": "http://gate.decodo.com:7000",
                "username": os.getenv("DECODO_PROXY_USERNAME"),
                "password": os.getenv("DECODO_PROXY_PASSWORD"),
            },
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/120.0.0.0 Safari/537.36"
            ),
        )
        page = await context.new_page()

        # block unnecessary resources
        await page.route(
            "**/*",
            lambda route: route.abort()
            if route.request.resource_type in ["image", "font", "stylesheet"]
            else route.continue_(),
        )

        await page.goto(url, wait_until="domcontentloaded")
        await page.wait_for_selector(".price_color", timeout=10000)
        await asyncio.sleep(random.uniform(2, 5))

        html = await page.content()
        await browser.close()

    products = parse_prices(html, url)
    print_summary(products)
    save_to_json(products)


asyncio.run(main())

import asyncio
import json
import os
import random
from datetime import datetime
from typing import Optional

from bs4 import BeautifulSoup
from dotenv import load_dotenv
from playwright.async_api import async_playwright
from pydantic import BaseModel

load_dotenv()


class ProductPrice(BaseModel):
    product_name: str
    price: float
    currency: str
    original_price: Optional[float] = None
    discount: Optional[float] = None
    availability: str
    source_url: str
    scraped_at: datetime = datetime.utcnow()


def parse_prices(html, source_url):
    soup = BeautifulSoup(html, "html.parser")
    products = []

    for article in soup.select("article.product_pod"):
        try:
            name = article.select_one("h3 a")["title"]
            raw_price = article.select_one(".price_color").text.strip()
            price = float(raw_price.replace("£", "").replace(",", ""))
            availability = article.select_one(".availability").text.strip()

            record = ProductPrice(
                product_name=name,
                price=price,
                currency="GBP",
                availability=availability,
                source_url=source_url,
            )
            products.append(record.model_dump())

        except (AttributeError, TypeError, ValueError):
            continue

    return products


def save_to_json(products):
    timestamp = datetime.utcnow().strftime("%Y%m%d_%H%M%S")
    filename = f"prices_{timestamp}.json"
    with open(filename, "w") as f:
        json.dump(products, f, indent=2, default=str)
    print(f"Saved {len(products)} products to {filename}")


def print_summary(products):
    print(f"\n{'Product':<50} {'Price':>10} {'Availability':<20}")
    print("-" * 82)
    for p in products[:10]:
        print(
            f"{p['product_name'][:48]:<50} "
            f"{p['currency']} {p['price']:>7.2f} "
            f"{p['availability']:<20}"
        )


async def main():
    url = "https://books.toscrape.com/catalogue/category/books_1/index.html"

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            proxy={
                "server": "http://gate.decodo.com:7000",
                "username": os.getenv("DECODO_PROXY_USERNAME"),
                "password": os.getenv("DECODO_PROXY_PASSWORD"),
            },
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/120.0.0.0 Safari/537.36"
            ),
        )
        page = await context.new_page()

        # block unnecessary resources
        await page.route(
            "**/*",
            lambda route: route.abort()
            if route.request.resource_type in ["image", "font", "stylesheet"]
            else route.continue_(),
        )

        await page.goto(url, wait_until="domcontentloaded")
        await page.wait_for_selector(".price_color", timeout=10000)
        await asyncio.sleep(random.uniform(2, 5))

        html = await page.content()
        await browser.close()

    products = parse_prices(html, url)
    print_summary(products)
    save_to_json(products)


asyncio.run(main())

For alternative output formats, CSV for spreadsheet analysis or SQLite for historical price tracking, see how to save scraped data for a full breakdown. If you'd rather skip building a custom scraper entirely, Decodo's eCommerce price scraper API returns structured pricing data without writing or maintaining any selector logic.

Testing your scraper with real results

Building the scraper is the first step. To make sure it works properly and to catch any errors, you need to check its results against the actual website. Run the scraper on books.toscrape.com and compare the output to what you see on the page.

Running the scraper

Make sure your virtual environment is active, then run:

source venv/bin/activate
python3 price_scraper.py

source venv/bin/activate
python3 price_scraper.py

You should see a summary table printed to the console immediately:

And a timestamped JSON file in your project directory:

[
 {
   "product_name": "A Light in the Attic",
   "price": 51.77,
   "currency": "GBP",
   "original_price": null,
   "discount": null,
   "availability": "In stock",
   "source_url": "https://books.toscrape.com/catalogue/category/books_1/index.html",
   "scraped_at": "2026-04-28 21:02:04.653382"
 },
  ...
]

[
 {
   "product_name": "A Light in the Attic",
   "price": 51.77,
   "currency": "GBP",
   "original_price": null,
   "discount": null,
   "availability": "In stock",
   "source_url": "https://books.toscrape.com/catalogue/category/books_1/index.html",
   "scraped_at": "2026-04-28 21:02:04.653382"
 },
  ...
]

Verifying accuracy

Open https://books.toscrape.com/catalogue/category/books1/index.html_ in your browser and manually check 5 products against your JSON output:

• Does the product name match exactly?
• Does the price match down to the decimal?
• Is the availability status correct?

A well-configured scraper should achieve 100% accuracy on books.toscrape.com. This site is simple and has no anti-bot measures, JavaScript-rendered content, or changing prices. If you notice any errors, the problem is with your selector logic, not the website itself.

Calculate your capture rate:

# add this to price_scraper.py to check coverage
def calculate_accuracy(products, expected_count):
    captured = len(products)
    accuracy = (captured / expected_count) * 100
    print(f"\nCapture rate: {captured}/{expected_count} products ({accuracy:.1f}%)")

# add this to price_scraper.py to check coverage
def calculate_accuracy(products, expected_count):
    captured = len(products)
    accuracy = (captured / expected_count) * 100
    print(f"\nCapture rate: {captured}/{expected_count} products ({accuracy:.1f}%)")

The website books.toscrape.com shows 20 products on each page. If you see fewer than 20 products, it means some are missing.

Diagnosing common failures

Zero products returned

The selector isn't finding anything. This might happen because the page didn't load completely before the parser started, or the HTML layout is different from what you expected. To troubleshoot, add a step to check what the scraper actually sees:

# temporarily add to main() after getting page content
with open("debug.html", "w") as f:
    f.write(html)

# temporarily add to main() after getting page content
with open("debug.html", "w") as f:
    f.write(html)

Open debug.html in your browser. If it looks different from the live page, such as missing products or incomplete HTML, it means the page did not finish loading before calling page.content(). To fix this, increase the timeout or use a more specific wait_for_selector command.

Prices returning as None or 0.0

The price element is in the HTML, but your selector isn't working. Open your browser's DevTools on the page you're checking. Right-click the price element and choose Inspect. Make sure the class name matches your selector exactly, as any small difference will cause it to fail.

Stale selectors after a site redesign

Websites often change their HTML structure unexpectedly. This means a selector that worked last week might not work today. This problem is known as selector rot, and it's the main reason why many scrapers stop working. To fix this issue, follow best practices: store selectors in a config file instead of hardcoding them in your script.

Incorrect prices from cached content

Some pages show older prices that may not be accurate. If a scraped price seems strange, either much lower or much higher than expected, make sure to check it for validation:

def validate_price(price, min_price=0.50, max_price=1000.00):
    if not min_price <= price <= max_price:
        print(f"Warning: price {price} outside expected range")
        return False
    return True

def validate_price(price, min_price=0.50, max_price=1000.00):
    if not min_price <= price <= max_price:
        print(f"Warning: price {price} outside expected range")
        return False
    return True

Flag outliers for review instead of simply discarding them; an exceptionally low price often signals a genuine sale rather than a processing error.

Why periodic test scrapes matter

A scraper that functions today offers no guarantee for next week. Websites frequently update their layouts, rotate class names, and deploy new anti-bot layers without warning. By running a test scrape against a known baseline, even on a weekly schedule, you can identify selector rot and breakage before they result in significant gaps in your collected data.

Set a simple check: if the number of products returned drops below a threshold, send an alert:

def check_scraper_health(products, minimum_expected=15):
    if len(products) < minimum_expected:
        print(f"Alert: only {len(products)} products returned. Scraper may be broken.")

def check_scraper_health(products, minimum_expected=15):
    if len(products) < minimum_expected:
        print(f"Alert: only {len(products)} products returned. Scraper may be broken.")

This won't catch every failure mode, but it catches the most common one: a selector breaking silently and returning an empty list.

Common obstacles and how to handle them

books.toscrape.com is easy to access. It doesn't have anti-bot measures, JavaScript rendering, or IP blocks. Real eCommerce sites will present challenges. Here’s what you can expect and how to deal with each issue.

CAPTCHAs and bot detection

Major retailers use aggressive methods to detect bots. They check your browser settings and look at how you make requests. If they see a single IP address making many requests to the same product pages quickly, it raises a red flag. They might ask you to complete a CAPTCHA if they suspect something is automated.

Three things reduce detection risk significantly:

• Rotate IPs. Decodo's residential proxies send each request through a unique residential IP address. This makes your scraper appear like regular traffic from various users
• Realistic browser fingerprints. Playwright already runs a real Chromium browser, which passes most fingerprint checks that headless detection targets. You can enhance this by setting a realistic user_agent and viewport size
• Request pacing. Adding random delays between requests is more important than many realize. Fixed delays are easy to detect, but random ones are not

For tough websites that use Cloudflare, Akamai, or custom bot protection, Decodo's Web Scraping API can handle fingerprinting, solve CAPTCHA, and JavaScript rendering as a managed layer. You provide a URL, and it returns the rendered HTML. For more details on how to bypass CAPTCHA, check out the guide that explains all the methods.

IP blocks and rate limiting

Getting blocked from a website isn’t always clear. You might see a 403 error. More often, the site returns a soft block, like a CAPTCHA page, an empty product page, or sends you to a login screen, while your scraper saves the incorrect HTML.

Here are some practical guidelines for most major retailers:

• Wait 5 to 10 seconds between requests to the same website.
• Limit to 1 or 2 active sessions per IP address.
• Change user agents along with IPs. Using the same user agent string for thousands of requests can identify your activity.

import random

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
]

def get_random_user_agent():
    return random.choice(USER_AGENTS)

import random

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
]

def get_random_user_agent():
    return random.choice(USER_AGENTS)

Dynamic content and JavaScript rendering

Prices loaded with AJAX or client-side rendering are not included in the initial HTML response. They only appear after JavaScript runs. When you make a basic Requests call, you get the page structure without any price data.

Playwright addresses this issue automatically. It uses a real browser and waits for the page to load completely before you access the content. However, for pages where prices show up only after user actions like scrolling, clicking a size selector, or accepting a cookie banner, you need to be more specific in your approach:

# wait for a specific price element to appear
await page.wait_for_selector("[data-testid='price']", timeout=15000)

# or wait for network activity to settle
await page.wait_for_load_state("networkidle")

# wait for a specific price element to appear
await page.wait_for_selector("[data-testid='price']", timeout=15000)

# or wait for network activity to settle
await page.wait_for_load_state("networkidle")

Use Playwright for pages that need JavaScript to load their content. For static pages where prices are already in the HTML, using requests with BeautifulSoup is faster and costs less. For more on what a headless browser is and when you need one, that covers the trade-offs. For more details on scraping dynamic content, refer to the deeper guide.

Anti-scraping countermeasures

Beyond CAPTCHAs and IP blocks, eCommerce sites use subtler traps:

• Honeypot links. Invisible links in the HTML that legitimate users never click, but scrapers follow all <a> tags will. Clicking one flags your session immediately. Filter links by visibility before following them:

# only follow visible links
links = await page.query_selector_all("a:visible")

# only follow visible links
links = await page.query_selector_all("a:visible")

• Obfuscated class names. dynamically generated class names like a3B9x_price that change on every deploy. Selector rot sets in fast when you target these; they break silently after every site update. Build selectors that rely on structural position or ARIA attributes instead:

# fragile -- breaks when class name changes
soup.select_one(".a3B9x_price")

# more resilient -- targets semantic role
soup.select_one("[aria-label='price']")
soup.select_one("[data-testid='product-price']")

# fragile -- breaks when class name changes
soup.select_one(".a3B9x_price")

# more resilient -- targets semantic role
soup.select_one("[aria-label='price']")
soup.select_one("[data-testid='product-price']")

• ARIA attributes (aria-label, aria-describedby, role) are set by developers for accessibility purposes. They're tied to the element's function, not its visual styling, which makes them far more stable across redesigns than class names.

For a full breakdown of anti-scraping techniques and how to handle them.

Geo-specific pricing

The same product can have different prices based on where you are. For example, a pair of headphones might cost $299 in the US, €319 in Germany, and £259 in the UK. These differences are not just due to currency conversion; they reflect actual pricing differences.

Use Decodo's geo-targeted residential proxies to scrape from specific regions:

# target a specific country via session parameter
proxy = {
    "server": "http://gate.decodo.com:7000",
    "username": f"{os.getenv('DECODO_PROXY_USERNAME')}-country-de",
    "password": os.getenv("DECODO_PROXY_PASSWORD")
}

# target a specific country via session parameter
proxy = {
    "server": "http://gate.decodo.com:7000",
    "username": f"{os.getenv('DECODO_PROXY_USERNAME')}-country-de",
    "password": os.getenv("DECODO_PROXY_PASSWORD")
}

Run the same scraper with different geographic targets and compare the results to understand pricing in different regions.

Data quality issues

Bad data can cause more problems than having no data at all. If a price seems valid but is incorrect, it can mess up any analysis without showing an error.

Here are some common sources of bad price data:

• Cached pages. Sometimes, a server sends an outdated version of a price from the previous day. Check scraped prices against a sensible range and mark any that are too far off
• Missing discount information. If the sale price is recorded but the original price is missing, you can't calculate the discount
• Currency mismatches. A price taken from a German website may be stored as if it were in USD, which leads to confusion

Add a validation layer before saving:

def validate_price(price, currency, min_price=0.01, max_price=50000.00):
    if not min_price <= price <= max_price:
        print(f"Warning: {currency} {price} outside expected range -- flagging for review")
        return False
    return True

def validate_price(price, currency, min_price=0.01, max_price=50000.00):
    if not min_price <= price <= max_price:
        print(f"Warning: {currency} {price} outside expected range -- flagging for review")
        return False
    return True

Flag outliers instead of throwing them away. An unusually low price might be a real flash sale, not an error from data scraping. To learn more about cleaning and preparing scraped data before using it, check out what is data cleaning?

Using AI to automate and optimize price scraping

Using manual selectors to write data works well when you're scraping information from one website. When you try to scrape data from 50 different retailers, each with its own HTML structure, class names, and page layout, this method doesn’t work effectively. AI can improve this process in several important ways.

AI-assisted selector generation

Instead of manually inspecting the DOM for every new target site, send the page HTML to an LLM, then ask the model to provide the CSS selector or XPath for the price element.

Here's a practical example using the Anthropic API. First, install Anthropic

pip install Anthropic SDK:

pip install anthropic

pip install anthropic

Create a new file, get_price_selector.py, and add the code below

import asyncio
import os
from dotenv import load_dotenv
from playwright.async_api import async_playwright
import anthropic

load_dotenv()

async def get_price_selector(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            proxy={
                "server": "http://gate.decodo.com:7000",
                "username": os.getenv("DECODO_PROXY_USERNAME"),
                "password": os.getenv("DECODO_PROXY_PASSWORD"),
            },
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/120.0.0.0 Safari/537.36"
            ),
        )
        page = await context.new_page()

        # wait for dynamic content to finish loading
        await page.goto(url, wait_until="networkidle")
        await asyncio.sleep(3)

        # strip junk and pass only the body
        html = await page.evaluate('''() => {
            const clone = document.body.cloneNode(true);
            clone.querySelectorAll('script, style, noscript, svg').forEach(el => el.remove());
            return clone.innerHTML;
        }''')
        await browser.close()

    html_sample = html[:50000]

    client = anthropic.Anthropic()

    message = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": f"""Analyze this HTML and return only the CSS selector
                for the product price element. Return nothing else -- just the selector string.
                No markdown, no code fences, no explanation.

                HTML:
                {html_sample}"""
            }
        ]
    )

    raw_selector = message.content[0].text.strip()

    # clean up common formatting artifacts
    selector = (
        raw_selector
        .replace("```css", "")
        .replace("`" * 3, "")
        .replace("css", "")
        .strip()
        .rstrip(".")
    )

    return selector

import asyncio
import os
from dotenv import load_dotenv
from playwright.async_api import async_playwright
import anthropic

load_dotenv()

async def get_price_selector(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            proxy={
                "server": "http://gate.decodo.com:7000",
                "username": os.getenv("DECODO_PROXY_USERNAME"),
                "password": os.getenv("DECODO_PROXY_PASSWORD"),
            },
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/120.0.0.0 Safari/537.36"
            ),
        )
        page = await context.new_page()

        # wait for dynamic content to finish loading
        await page.goto(url, wait_until="networkidle")
        await asyncio.sleep(3)

        # strip junk and pass only the body
        html = await page.evaluate('''() => {
            const clone = document.body.cloneNode(true);
            clone.querySelectorAll('script, style, noscript, svg').forEach(el => el.remove());
            return clone.innerHTML;
        }''')
        await browser.close()

    html_sample = html[:50000]

    client = anthropic.Anthropic()

    message = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": f"""Analyze this HTML and return only the CSS selector
                for the product price element. Return nothing else -- just the selector string.
                No markdown, no code fences, no explanation.

                HTML:
                {html_sample}"""
            }
        ]
    )

    raw_selector = message.content[0].text.strip()

    # clean up common formatting artifacts
    selector = (
        raw_selector
        .replace("```css", "")
        .replace("`" * 3, "")
        .replace("css", "")
        .strip()
        .rstrip(".")
    )

    return selector

Cache the returned selector you get in a config file. Use it for each scrape until it stops giving results. Then, create a new one. This way, you let AI do the hard discovery work only once, not with every request.

LLM-based extraction from unstructured pages

Some product pages have prices that are hard to find because they use different types of HTML or plain text. This makes it difficult to target with CSS selectors. can read these pages and provide the information in a clear JSON format without needing any selectors.

def normalize_prices_with_llm(raw_prices: list) -> list:
    client = anthropic.Anthropic()
    
    # We pre-fill the assistant response with '[' to lock it into a JSON array
    message = client.messages.create(
        model="claude-haiku-4-5-20251001", 
        max_tokens=1024,
        system="You're a data normalization expert. Return ONLY valid JSON arrays.",
        messages=[
            {
                "role": "user",
                "content": f"Normalize to JSON (amount: float, currency: ISO 4217): {raw_prices}"
            },
            {
                "role": "assistant",
                "content": "[" # This 'pre-fill' trick forces JSON output
            }
        ]
    )
    
    import json
    # Re-attach the bracket we forced
    full_json = "[" + message.content[0].text.strip()
    return json.loads(full_json)

def normalize_prices_with_llm(raw_prices: list) -> list:
    client = anthropic.Anthropic()
    
    # We pre-fill the assistant response with '[' to lock it into a JSON array
    message = client.messages.create(
        model="claude-haiku-4-5-20251001", 
        max_tokens=1024,
        system="You're a data normalization expert. Return ONLY valid JSON arrays.",
        messages=[
            {
                "role": "user",
                "content": f"Normalize to JSON (amount: float, currency: ISO 4217): {raw_prices}"
            },
            {
                "role": "assistant",
                "content": "[" # This 'pre-fill' trick forces JSON output
            }
        ]
    )
    
    import json
    # Re-attach the bracket we forced
    full_json = "[" + message.content[0].text.strip()
    return json.loads(full_json)