Cloudflare Scraper: Which Tools Still Work in 2026 and How To Build One

TL;DR

• A Cloudflare scraper is a tool that extracts data from sites with Cloudflare's protection
• Cloudflare scores every request based on the TLS fingerprint, JavaScript rendering, IP reputation, and behavior, so no single fix really helps bypass it
• Many tools available today are outdated or tackle specific parts of Cloudflare. The most practical approach is to use curl_cffi to handle TLS-only blocks, but anything requiring JavaScript needs a maintained patched headless browser
• The most common pitfall is assuming a 200 response is proof of success because a challenge page returns one with normal-looking HTML that parses to nothing
• Switch to a managed API when maintenance stops making financial or practical sense

What is a Cloudflare scraper?

A Cloudflare scraper is any tool designed to extract data from websites sitting behind Cloudflare's protection. It usually comes in one of 3 forms: an open-source library, a patched browser, or a managed API.

Think of Cloudflare as a bouncer at a club. It checks who you are, how you move, and whether you look like you belong. If anything feels off, you're out.

Standard HTTP libraries like Requests or curl walk up to that bouncer wearing sneakers at a black-tie event. They don't run JavaScript. They don't match real browser fingerprints. They get turned away at the door.

One quick note if you're here for a specific package: cloudscraper isn't a generic term, it's a distinct Python library built for this exact job, and we cover it in our cloudscraper guide.

If you're new to this field, start with our intro to what is web scraping.

Cloudflare scraper tools compared

While some of these might be outdated in 2026, this table sorts each option by where it stopped keeping pace, so you can rule out the ones that will waste your time before you install anything. If you want the lightest possible option, start with an HTTP client that sends a browser-grade handshake, the same idea behind scraping with curl.

If it's hard to decide, here's a quick cheat sheet:

• If you're only being blocked at the handshake, use curl_cffi as it's the cheapest option on the market
• If the page needs JavaScript to load, use a supported browser-based option like CDP Mode in SeleniumBase or nodriver
• If maintaining your own stack is just not viable anymore, move to a managed API

What is Cloudflare and how does it block scrapers?

Cloudflare is a CDN (the network that speeds up websites by caching them globally) and security service sitting between users and websites. When you send a request to a protected site, Cloudflare inspects you first, scores you, and then decides whether to let you through.

That scoring system is called Bot Management, and it runs on multiple detection layers at once.

TLS fingerprinting

TLS (Transport Layer Security) is the handshake that happens when your client connects over HTTP(S). Every browser has its own TLS signature, which includes an order of ciphers it supports, the extensions it uses, and so on. You'll see this signature referred to as a JA3 or JA4 fingerprint in most tooling docs, JA4 being the newer format built to tackle the randomization modern browsers now apply to TLS extension order. Cloudflare checks that signature against known browser fingerprints. So, if your Python script sends a TLS handshake that doesn't match Chrome, Firefox, or Safari, it stands out.

JavaScript challenges

Cloudflare serves small JavaScript puzzles to every visitor. Real browsers solve them silently in the background. Scripts using Requests can't because they don't run JavaScript at all. No solution, no access.

For more on handling JS-heavy pages, see our guide on how to web scrape dynamic content.

Behavioral analysis

Cloudflare watches how you browse. Do you jump straight to a product page with no homepage visit? Do you click faster than humanly possible? Do you request 500 pages in 10 seconds? All red flags.

Turnstile CAPTCHAs

Turnstile is Cloudflare's replacement for the classic "I'm not a robot" checkbox. It runs behind the scenes, testing browser capabilities without bothering the user. If your scraper can't pass, you'll see a challenge page instead of the data you wanted.

When any of these checks fail, you get 1 of 3 things back: a challenge page, an error code, or a full IP ban. The specific code tells you which layer tripped, from the network up to your request pattern, so it's worth checking against the error code table further down before you start debugging blind.

VM-based challenges

Cloudflare's current challenge script does more than just run a puzzle. It practically delivers a small virtual machine written in JavaScript. The real logic arrives as bytecode that only that interpreter understands, and both the VM and the bytecode rotate regularly.

This is what separates the working tools from the dead ones. Libraries like cloudscraper pass challenges by reimplementing the math in Python, which works when the challenge is a readable equation and fails when it's an obfuscated program running inside a browser. There's nothing left to port, so the only reliable way through is to execute it in a real browser engine.

CDP Mode

CDP (Chrome DevTools Protocol) is the channel automation tools use to drive Chrome. Cloudflare checks for the traces automation leaves behind, and a browser driven through the standard WebDriver path is detectable before any of your stealth patches run.

CDP Mode means controlling the browser through the DevTools Protocol directly instead, skipping the WebDriver layer and the fingerprints that come with it. SeleniumBase offers this as a mode you switch on, which is why the table above lists UC Mode on its own as insufficient. CDP isn't invisible either as enabling certain protocol domains leaves artifacts of its own, which is exactly what rebrowser-patches is here to suppress.

Why Cloudflare blocks scrapers and common mistakes developers make

Most failed scrapers don't get caught because Cloudflare is magic. They get caught because they make the same small mistakes over and over. Let's walk through them.

Why Cloudflare blocks you

• Your browser fingerprint is missing or doesn't match a real browser
• Your scraper can't execute JavaScript, so challenge scripts go unsolved
• Your IP is suspicious, for example, datacenter IPs, public proxy ranges, and blocklisted addresses light up immediately
• Your request pattern looks robotic, meaning it’s too fast, too uniform, there's no homepage visit

Common developer mistakes

• Using Python Requests or curl alone. They're great tools, but they can't solve JS challenges or pass fingerprint checks. It's like showing up to a chess tournament with checkers pieces.
• Copying static headers from your browser into your script. Headers alone won't get the job done. Cloudflare looks at a long list of signals all at once, and headers are just one of them.
• Using free or public proxies. These are the cheap hotels of the proxy world. Everyone's stayed there, and the place is already identified. Many are also insecure, so you're risking your own data. See our guide on how to test proxies before trusting any.
• Hammering a single IP at full speed. Rate limits kick in fast, and once you're flagged, you're stuck. If you hit timeouts, our post on Python requests retry covers graceful backoff.
• Ignoring cookies. Cloudflare drops a cf_clearance cookie once you pass its checks. Throw it away, and you'll have to re-verify on every request.
• Jumping straight to the data page. Real users visit the homepage first, load assets, maybe click around. Skipping that warm-up is like walking into a meeting halfway through and starting to talk.
• Treating a 200 response as success. A Cloudflare challenge page can arrive wrapped in a 200 response, especially through an API or headless browser. It's the most common silent failure that runs cleanly for weeks while quietly storing nothing useful. Check the page content, not just the status code.

Troubleshooting Cloudflare error codes

All these codes can be split into 2 groups: the 1xxx codes are Cloudflare's own verdict on your request and appear in the HTML body of the response, while the HTTP codes arrive in the status header, and the 52x range describes a fault between Cloudflare and the site's origin server.

Techniques for CAPTCHA solving

Getting past Cloudflare means solving CAPTCHAs. But Turnstile runs silently and needs a full browser environment to pass. Plain scripts can't touch it, so your options are:

• Use a headless browser with stealth patches. Keep in mind that Turnstile needs a genuine browser environment plus a clean IP reputation, and stealth patching alone doesn't reliably clear it. It's the least reliable of the 3 options.
• Plug in a third-party CAPTCHA solver like2Captchaor CapSolver. They charge per solve, but they work when automation hits a wall. For the general strategy, see how to bypass Google CAPTCHA.
• Use a scraping API that handles JavaScript rendering and CAPTCHA solving in 1 call. This is where most production scrapers end up – the math on maintenance time usually favors an API.

Advanced tips: Rotating proxies and geotargeting

Once you've got JavaScript and CAPTCHAs under control, your next battleground is IP reputation. Cloudflare bans by IP as fast as it bans by fingerprint. Here's how to stay ahead.

Proxy rotation

Rotating proxies spread your requests across many IPs, so no single address gets hammered into a ban. Think of it like a relay race – each runner handles a short leg, then passes off before getting tired. Our guide on rotating proxies goes deeper.

• Residential proxies. These are real IPs from real households. To Cloudflare, these look exactly like regular users on home internet. Highest success rate, and the top pick for serious Cloudflare work. Decodo's residential proxies pool has 115M+ IPs from real households worldwide.
• Datacenter proxies. They're faster and cheaper, but Cloudflare spots datacenter IP ranges easily. Fine for sites with light protection, risky for heavily guarded ones.
• ISP proxies. Static residential IPs hosted on ISP infrastructure. You get the speed of a datacenter and the legitimacy of residential in one. Great for session-based scraping where you need the same IP across multiple requests. See Decodo ISP proxies.

For more on the residential vs. datacenter split, see what is a residential proxy network. If you need a new IP on every connection, our post on how to generate a random IP address breaks that down.

Geotargeting

Some Cloudflare rules are region-specific. A request from São Paulo might face a harder challenge than one from New York for the same US site. A few quick rules:

• Match your proxy country to the site's primary audience. Scraping a US retailer? Use US IPs
• Use geotargeting to access content that's only visible in certain countries
• Decodo's rotating proxies support country, state, and city-level targeting

Session management

Sticky sessions let you keep the same IP across several requests. That matters when you're logged in, adding items to a cart, or scraping paginated results – breaking the session breaks your cookies.

Warm up your sessions. Visit the homepage first, load some assets, accept the cookie banner, then go to the data page. Skipping this looks unnatural. It's the difference between walking into a coffee shop and ordering, versus sprinting past the counter straight to the pastry case.

Scraping Cloudflare-protected sites with Decodo Web Data API

We'll scrape Crunchbase, a company intelligence platform sitting behind Cloudflare. Our target is Slack's company profile, where we'll pull the description, headquarters, founders, leadership, and investor list. We chose Crunchbase because it's a real, high-traffic, Cloudflare-protected site that publishes structured data genuinely useful for market research, lead generation, and competitor tracking.

Get Decodo Web Data API

Sign up for a Decodo dashboard account and claim your free plan, so you can test the tool before committing. From your dashboard, grab the Web Data API token and store it in a .env file.

Set up your Python environment

You'll need Python 3.8+ and 3 libraries: Requests for HTTP calls, python-dotenv for loading your .env file, and Beautiful Soup for parsing HTML. Install them with pip:

pip install requests python-dotenv beautifulsoup4

Now create 2 files in your project folder:

cloudflare-scraper/
├── .env
└── scraper.py

Add your API basic authentication token to an .env file:

DECODO_TOKEN=your_api_token_here

Make your first API request

The Decodo Web Data API takes your target URL and runs it through a real browser. JavaScript rendering and Cloudflare bypass happen automatically. You get back the fully rendered HTML, and proxies, fingerprints, and CAPTCHAs are all handled server-side.

Heads up on timing, though. A Cloudflare-protected page with JavaScript rendering turned on isn't instant. Give your request a generous timeout, since 15 to 20 seconds of round-trip time is normal for these pages.

We'll build the script in 5 small pieces, then show the full thing at the end. The target is Slack's Crunchbase profile, and we'll extract the company details, founders, leadership, and investors.

Imports and configuration

Open your scraper.py file and start with the imports, the token, and the 2 URLs. The token comes from .env via python-dotenv, which keeps your credentials out of source control.

from __future__ import annotations

import json
import os

import requests
from bs4 import BeautifulSoup
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

TOKEN = os.getenv("DECODO_TOKEN")
TARGET_URL = "https://www.crunchbase.com/organization/slack"
SCRAPING_URL = "https://data.decodo.com/v1/scrape"

AUTH_HEADERS = {
    "Authorization": f"Bearer {TOKEN}",
    "Content-Type": "application/json",
    "Accept": "application/json",
}

3 notes to keep in mind:

• SCRAPING_URL is Decodo's API endpoint where we POST our request, not the Crunchbase URL directly
• AUTH_HEADERS uses Bearer {TOKEN} as the authorization value and the API treats this as your credentials
• json comes from the standard library, and we'll need it to read Crunchbase's structured data later

The scraping function

Next, a function that sends the request and handles whatever comes back. This function covers the 5 main cases: a successful response, empty results, HTTP errors, network timeouts, and anything else that might crash. Your calling code only ever deals with clean strings or dicts.

def scrape_with_decodo(url: str) -> str | dict:
    try:
        response = requests.post(
            SCRAPING_URL,
            headers=AUTH_HEADERS,
            json={
                "url": url,
                "headless": "html",
                "geo": "United States",
            },
            timeout=60,
        )
        # 1. Automatically raises HTTPError for 4xx or 5xx codes
        response.raise_for_status()

        # 2. Process successful 200 response
        data = response.json()
        results = data.get("results", [])

        # 3. A failed job returns no results, but Decodo explains why
        if not results:
            return {
                "status_code": data.get("status_code"),
                "message": data.get("message", "No results returned"),
            }

        return results[0].get("content", "")

    except requests.exceptions.HTTPError:
        # 4. Handle specific status codes routed here by raise_for_status()
        if response.status_code == 401:
            return "Unauthorised - check your API token"
        if response.status_code == 403:
            return "Access denied - target may be blocked"
        return f"HTTP error occurred: {response.status_code} - {response.text}"

    except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as exc:
        # 5. Handle network-level issues
        return f"Network error: {exc}"

    except Exception as exc:
        # 6. Catch-all for JSON parsing or other logic errors
        return f"Unexpected error: {exc}"

There are 3 parameters in the JSON body that do most of the work:

• url – the target page you want rendered
• "headless": "html" – renders the page in a real browser and hands back the final HTML
• "geo": "United States" – sends the request from a US residential IP

Any 4xx or 5xx response jumps straight into the HTTPError branch, so we can react to specific codes. If your token is wrong, you'll get 401, and if Decodo reached the page but got blocked anyway (rare), you'll get 403. A job can also fail without an HTTP error, in which case Decodo returns no results along with its own status code and message explaining what went wrong. Pass that through instead of guessing, since a generic "check your token" message sends you debugging the wrong thing.

Parsing the HTML

Getting the HTML back is half the job. The other half is pulling the company details out of it. Crunchbase publishes its profile data as JSON-LD, the structured data format search engines read, inside script tags. That's a better parsing target than CSS classes or DOM structure, because Crunchbase maintains it deliberately for search engines, and it survives redesigns that would break a selector.

def extract_company(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")
    company = {}

    for block in soup.select('script[type="application/ld+json"]'):
        try:
            data = json.loads(block.get_text())
        except ValueError:
            # Skip malformed blocks rather than stopping the whole parse
            continue

        entity = data.get("mainEntity", {})

        # The company profile lives in the Corporation entity
        if isinstance(entity, dict) and entity.get("@type") == "Corporation":
            address = entity.get("address", {})
            company.update({
                "name": entity.get("name"),
                "description": entity.get("description"),
                "ticker": entity.get("tickerSymbol"),
                "headquarters": ", ".join(
                    part for part in (
                        address.get("addressLocality"),
                        address.get("addressRegion"),
                        address.get("addressCountry"),
                    ) if part
                ),
                "founders": [p.get("name") for p in entity.get("founder", [])],
                "executives": [
                    f"{p.get('name')} ({p.get('jobTitle')})"
                    for p in entity.get("employee", [])
                ],
                "investors": [o.get("name") for o in entity.get("funder", [])],
            })

        # Crunchbase also publishes a FAQ block with funding and competitor facts
        if data.get("@type") == "FAQPage":
            company["faq"] = [
                {
                    "question": q.get("name"),
                    "answer": q.get("acceptedAnswer", {}).get("text"),
                }
                for q in data.get("mainEntity", [])
            ]

    return company

Inside this function:

• soup.select('script[type="application/ld+json"]') finds every structured data block on the page, and a company profile carries 4 of them
• The profile sits in the block whose mainEntity is a Corporation, holding the name, description, ticker symbol, address, founders, employees, and funders
• The FAQPage block is separate, carrying Crunchbase's own answers on headquarters, investor count, last funding round, and competitors
• A malformed block gets skipped rather than crashing the run, since 1 bad script tag shouldn't cost you the whole page

Keep in mind that Crunchbase blanks out funding amounts for anonymous visitors, returning the literal string "obfuscated" instead of the real figure, since monetary data sits behind a paid plan

Before parsing, run the response through a quick check for challenge markers. This catches the silent failure described earlier, where a 200 response arrives carrying a challenge page instead of your content.

def looks_like_challenge(html: str) -> bool:
    # A real Crunchbase profile runs to roughly 1 MB. A challenge page is a
    # small static shell around a script, so size alone catches a lot of cases.
    too_small = len(html) < 100_000

    # Cloudflare prints a visible "Ray ID:" line only on its own interstitial
    # and error pages. The cf-ray header appears on every request, challenge
    # or not, but this text in the body means Cloudflare handled the request
    # itself instead of passing it to Crunchbase.
    has_ray_id = "Ray ID:" in html

    # A Turnstile challenge embeds its widget script directly in the page.
    # Finding it means you're looking at a task for the visitor to complete,
    # not the content you requested.
    has_turnstile = "challenges.cloudflare.com/turnstile" in html

    return too_small or has_ray_id or has_turnstile

That character count is 1 part of the check, not the whole thing. A tiny response usually means a challenge slipped through, but a page can also fail at full size, padded out by the challenge script itself. The Ray ID and Turnstile checks catch that case, so treat all 3 as 1 test rather than relying on size alone.

Running it

Now we wire it all together at the bottom of the file – scrape, verify we got real content, parse, and print the company profile.

if __name__ == "__main__":
    result = scrape_with_decodo(TARGET_URL)

    if not isinstance(result, str):
        print(f"Scrape failed: {result}")
    elif looks_like_challenge(result):
        print("Got a challenge page instead of content. Retry or adjust your settings.")
    else:
        print(f"\nRetrieved {len(result):,} characters of HTML\n")

        company = extract_company(result)

        if not company.get("name"):
            print("No company data found. Crunchbase may have changed its markup.")
        else:
            print(f"{company['name']} ({company['ticker']})")
            print(f"Headquarters: {company['headquarters']}")
            print(f"\n{company['description']}\n")

            print(f"Founders: {', '.join(company['founders'])}\n")

            print("Leadership:")
            for person in company["executives"][:5]:
                print(f"   {person}")

            print(f"\nInvestors: {', '.join(company['investors'][:5])}")

Run it with:

python scraper.py

The full script

Here's everything glued together, ready to copy into scraper.py:

from __future__ import annotations

import json
import os

import requests
from bs4 import BeautifulSoup
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

TOKEN = os.getenv("DECODO_TOKEN")
TARGET_URL = "https://www.crunchbase.com/organization/slack"
SCRAPING_URL = "https://data.decodo.com/v1/scrape"

AUTH_HEADERS = {
    "Authorization": f"Bearer {TOKEN}",
    "Content-Type": "application/json",
    "Accept": "application/json",
}

def scrape_with_decodo(url: str) -> str | dict:
    try:
        response = requests.post(
            SCRAPING_URL,
            headers=AUTH_HEADERS,
            json={
                "url": url,
                "headless": "html",
                "geo": "United States",
            },
            timeout=60,
        )
        # 1. Automatically raises HTTPError for 4xx or 5xx codes
        response.raise_for_status()

        # 2. Process successful 200 response
        data = response.json()
        results = data.get("results", [])

        # 3. A failed job returns no results, but Decodo explains why
        if not results:
            return {
                "status_code": data.get("status_code"),
                "message": data.get("message", "No results returned"),
            }

        return results[0].get("content", "")

    except requests.exceptions.HTTPError:
        # 4. Handle specific status codes routed here by raise_for_status()
        if response.status_code == 401:
            return "Unauthorised - check your API token"
        if response.status_code == 403:
            return "Access denied - target may be blocked"
        return f"HTTP error occurred: {response.status_code} - {response.text}"

    except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as exc:
        # 5. Handle network-level issues
        return f"Network error: {exc}"

    except Exception as exc:
        # 6. Catch-all for JSON parsing or other logic errors
        return f"Unexpected error: {exc}"

def extract_company(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")
    company = {}

    for block in soup.select('script[type="application/ld+json"]'):
        try:
            data = json.loads(block.get_text())
        except ValueError:
            # Skip malformed blocks rather than stopping the whole parse
            continue

        entity = data.get("mainEntity", {})

        # The company profile lives in the Corporation entity
        if isinstance(entity, dict) and entity.get("@type") == "Corporation":
            address = entity.get("address", {})
            company.update({
                "name": entity.get("name"),
                "description": entity.get("description"),
                "ticker": entity.get("tickerSymbol"),
                "headquarters": ", ".join(
                    part for part in (
                        address.get("addressLocality"),
                        address.get("addressRegion"),
                        address.get("addressCountry"),
                    ) if part
                ),
                "founders": [p.get("name") for p in entity.get("founder", [])],
                "executives": [
                    f"{p.get('name')} ({p.get('jobTitle')})"
                    for p in entity.get("employee", [])
                ],
                "investors": [o.get("name") for o in entity.get("funder", [])],
            })

        # Crunchbase also publishes a FAQ block with funding and competitor facts
        if data.get("@type") == "FAQPage":
            company["faq"] = [
                {
                    "question": q.get("name"),
                    "answer": q.get("acceptedAnswer", {}).get("text"),
                }
                for q in data.get("mainEntity", [])
            ]

    return company

def looks_like_challenge(html: str) -> bool:
    # A real Crunchbase profile runs to roughly 1 MB. A challenge page is a
    # small static shell around a script, so size alone catches a lot of cases.
    too_small = len(html) < 100_000

    # Cloudflare prints a visible "Ray ID:" line only on its own interstitial
    # and error pages. The cf-ray header appears on every request, challenge
    # or not, but this text in the body means Cloudflare handled the request
    # itself instead of passing it to Crunchbase.
    has_ray_id = "Ray ID:" in html

    # A Turnstile challenge embeds its widget script directly in the page.
    # Finding it means you're looking at a task for the visitor to complete,
    # not the content you requested.
    has_turnstile = "challenges.cloudflare.com/turnstile" in html

    return too_small or has_ray_id or has_turnstile

if __name__ == "__main__":
    result = scrape_with_decodo(TARGET_URL)

    if not isinstance(result, str):
        print(f"Scrape failed: {result}")
    elif looks_like_challenge(result):
        print("Got a challenge page instead of content. Adjust your token or parameters and try again.")
    else:
        print(f"\nRetrieved {len(result):,} characters of HTML\n")

        company = extract_company(result)

        if not company.get("name"):
            print("No company data found. Crunchbase may have changed its markup.")
        else:
            print(f"{company['name']} ({company['ticker']})")
            print(f"Headquarters: {company['headquarters']}")
            print(f"\n{company['description']}\n")

            print(f"Founders: {', '.join(company['founders'])}\n")

            print("Leadership:")
            for person in company["executives"][:5]:
                print(f"   {person}")

            print(f"\nInvestors: {', '.join(company['investors'][:5])}")

Example output

A successful run prints output like this:

Retrieved 1,172,027 characters of HTML

Slack (NYSE:WORK)
Headquarters: San Francisco, California, United States

Slack is a cloud-based communication and collaboration platform for teams.

Founders: Cal Henderson, Eric Costello, Serguei Mourachov, Stewart Butterfield

Leadership:
   Cal Henderson (Co-Founder and CTO)
   Eric Costello (Co-founder)
   Serguei Mourachov (Co-Founder and Principal engineer)
   Noah Weiss (VP of Product)
   Julie Liegl (CMO)

Investors: Light Street Capital, Meyer Ventures, Employee Stock Option Fund, General Atlantic, SoftBank Vision Fund

From here, you can save to CSV, push to a database, or feed into a dashboard. Our guide on how to save your scraped data covers the common output formats. And if you're new to Requests and Beautiful Soup, our primers on mastering Python requests and Beautiful Soup web scraping cover the fundamentals.

Keep in mind that JavaScript rendering is the line item that makes a Cloudflare-protected page cost more to fetch than a plain one, not the proxy itself. With Web Data API, a standard proxy request with no JS starts at $0.50/1K requests on the entry plan, while switching on JS rendering brings that to $0.75/1K, and to $1.50/1K if the target also needs premium proxies for a cleaner IP reputation.

If you need something even more hands-off, Decodo Site Unblocker handles anti-bot measures, CAPTCHAs, and fingerprinting automatically.

Best Cloudflare scraping practices

A scraper that works today but breaks next Tuesday isn't much use. These habits keep things running long-term:

• Rotate user agents per session, not per request, so fingerprints stay consistent within a session
• Add randomized delays between requests – 2 to 5 seconds minimum, as uniform timing is a classic bot tell
• Use residential proxies for heavily protected sites because the cost premium pays for itself in success rate
• Watch response sizes – a response suspiciously smaller than usual means it’s a challenge page
• Start with a single URL, confirm it works, then scale up
• Log every status code and response length for easier troubleshooting later
• Respect robots.txt and the site's Terms of Service – scrape the public stuff, skip the login-gated stuff, and don't overload servers
• Monitor for policy changes on top of detection changes. From 1 July 2026, Cloudflare lets a site declare what Search, Agent, and Training crawlers are permitted to do, and a bot can lose its Verified status for ignoring that.

What Cloudflare's 2026 AI crawler rules mean for your scraper

On 1 July 2026, Cloudflare expanded its single "Block AI bots" toggle into 3 declared categories: Search, Agent, and Training. Search covers anything indexing your content to answer questions about it later, Agent covers automated behavior acting in real time on a person's behalf, and Training covers crawling that feeds content into a model permanently. The controls reach every customer, free tier included. Since 15 September 2026, new domains onboarding to Cloudflare get Training and Agent blocked by default on pages that display ads, while Search stays allowed.

Cloudflare's reasoning is that an ad signals a page built for human attention, so the crawlers least likely to deliver a human get blocked there. A second change lands the same day for existing customers: multi-purpose crawlers that combine Search with Training are now judged on all their behaviors at once, so anyone who has chosen to block Training will also start blocking Googlebot, Applebot, and BingBot unless they opt out first.

Alongside this, Cloudflare has moved past per-crawl billing toward charging per use. Its Monetization Gateway lets site owners charge any caller for any resource behind Cloudflare, from a web page to an API to an MCP tool call, with metering, payment, and settlement handled at the edge instead of the origin. Blocking is no longer the only option a publisher has.

What this changes, and what it doesn't

Note that this is a policy layer sitting on top of bot detection, and nothing is truly replaced.

What changes is that Declared AI crawlers are affected by category. A crawler that identifies itself and holds Verified status now gets access based on what it does, not simply on being recognized. The same goes for the preferences a site publishes in its robots.txt.

What doesn't change is undeclared automated traffic meeting the same bot management stack it always has. Fingerprinting, challenges, IP reputation, and rate limits are untouched by any of this. A scraper that was getting through last week isn't newly blocked because of a category it never declared itself into.

In any case, getting this wrong in either direction is costly. Treating the category rules as a new wall in front of all automation overstates them, and ignoring them entirely misses where the ground is shifting.

Why this matters if you're building agents

An AI agent fetching a page on a user's behalf now sits in a named category that a growing share of the web blocks by default, even though there's a real person waiting on the other end of that request. Identifying yourself honestly, which used to be the cooperative move, now places you in a bucket with a default that may not favor you.

That turns reliable access to protected pages into an infrastructure decision rather than a scripting one. If your product depends on reading pages that sit behind Cloudflare, the question stops being which library patches which fingerprint and starts being what your access layer looks like: whether you're going through an MCP server, a managed API, or something you maintain yourself. Cloudflare has also been building traps specifically for crawlers that ignore these signals, which is worth reading up on separately in our guide to Cloudflare's AI Labyrinth. The broader picture of how this reshapes web data for AI is moving quickly enough that it deserves its own watch list.

Final thoughts

Cloudflare's protection is tough but not unbeatable. What it needs is the right mix of JavaScript rendering, realistic TLS fingerprints, clean residential IPs, and sensible session handling.

If you enjoy this kind of plumbing, you can wire all of that together yourself with Playwright, stealth plugins, proxy pools, and CAPTCHA solvers. But if you'd rather spend your time on the data itself, a dedicated scraping API removes the moving parts.

The switch point isn't scale, but rather maintenance frequency. A tool that breaks on every Cloudflare update costs more in engineering time than an API costs in credits. If you want to weigh that trade-off in more depth, our full breakdown of how to bypass Cloudflare covers it.

Whichever route you pick, scrape responsibly. Respect the site's rate limits, skip the sensitive stuff, and follow the law in your jurisdiction.