Web Scraping CLI for AI Agents: 6 Tools Tested

TL;DR

• Recommended for unattended jobs. pobear fetched directly for free, and --fail-on-empty made it report every empty page as a failure. With pobear's --ladder, our Web Data API returned every blocked page in the test.
• JavaScript. pobear and the Decodo CLI rendered the JavaScript page through the Web Data API. Crawl4AI (crwl) and keyless Firecrawl rendered it for free.
• Accounts. Of the 6 tools, 5 start without an account or API key, and the Decodo CLI has a free plan.
• Blocks. Of the 4 local tools (pobear, crwl, Trafilatura, and curl + htmlq), only pobear exited non-zero on every blocked page.

Pick a tool by scenario

The right choice depends on who reads the output. When a job runs on a schedule with no model reading it, the criterion is how clearly the tool reports a failure. When an agent writes a one-time scraper with you, the criterion is how many turns the agent needs to reach a correct result:

Coding agents such as Claude Code, Codex, and Cursor already call shell commands, so any of these tools works as a subprocess with no plugin. If you're building a scraping agent in Claude, start with our guide to scraping the web with Claude.

For workflows outside the command line, our list of the best web scraping tools compares APIs, libraries, and no-code options.

What makes a web scraping CLI agent-ready

An agent has to run a command-line tool (CLI), parse its output, and decide what to do next without reading human-readable messages. A CLI that follows 3 conventions makes these steps reliable:

• Data goes to stdout. stdout is the stream where a program prints its results. Results print as JSON, or as NDJSON (newline-delimited JSON, 1 object per line) that the agent can parse line by line.
• Logs and errors go to stderr. stderr is a separate output stream, so warnings don't mix with the data.
• Exit codes have documented meanings. An exit code is the number a process returns when it finishes, where 0 means success. A documented non-zero value tells the agent what went wrong.

A CLI can also write to files, pipe output to other commands, and run in any agent that has a shell, which is why agents often work better with a CLI than an MCP server. Chat assistants without a shell get the same data through an MCP (Model Context Protocol) server.

Printing only the requested fields keeps the agent's context small. It also sends less of the page to the model, which matters because a fetched page is untrusted input that can hide instructions (prompt injection).

The test distinguishes a hard block, an HTTP error such as 403 Forbidden, from a soft block, a challenge or CAPTCHA page served with HTTP 200 OK. A soft block appears in the page content, not in the HTTP status.

The test checked each tool against 6 criteria:

How we tested 6 scraping CLIs

Every tool got the same task. Each tool had to fetch the page, return the requested fields, and fail if nothing was extracted. Each tool ran as a subprocess, the way an agent calls it, and a test script recorded the command, exit code, output, and fields found. Each tool ran 3 times on each page.

The test used 4 pages:

JavaScript rendering means running a page's scripts in a browser engine before reading the HTML, which our dynamic rendering glossary entry covers in more detail. You can also read more in-depth on avoiding Amazon CAPTCHAs.

The values in parentheses are CSS selectors, the patterns that select elements on a page. The Decodo CLI, the Firecrawl CLI, and Trafilatura return Markdown or JSON instead of selector matches, so for those tools a field counted as found when its exact expected value appeared in the output. Indeed needed its own rule, because its listings change between runs. For every tool, an Indeed run counted as returning listings when its output contained at least 5 job-title keywords, such as Python developer, and no challenge page.

The JavaScript page embeds its quotes as a JSON array in an inline script, so a tool that reads script contents could get the data without rendering. The test counted only the rendered .quote elements, the way a selector-based agent reads the page.

These rules applied to every run:

• Defaults first. Every tool used default settings, unless a flag is the documented way to do the task, such as a rendering flag.
• Free and paid runs separated. Free modes ran with no API key set. Runs through the Web Data API (pobear with --provider decodo, and the Decodo CLI) ran separately.
• Same conditions within each comparison. Every test ran on macOS 27 on Apple Silicon over 1 home connection.
• Versions. pobear 0.3.0, Decodo CLI 3.0.0, Firecrawl CLI 1.26.22, Crawl4AI 0.9.5, Trafilatura 2.3.1, htmlq 0.4.0, curl 8.7.1, Claude Code 2.1.216 for the agent test, and jq 1.8.2 and check-jsonschema 0.38.2 for the scrape.sh script. The results and the cited docs describe these versions.

The test also covered 4 more scenarios:

• An agent's first try. A new Claude Code session got only a tool name and 1 task: to save the 10 quotes and authors from https://quotes.toscrape.com/scroll to quotes.json. Each session had to use its assigned tool (pobear, crwl, or the Firecrawl CLI) for every web request. Sessions ran headless, 3 times per tool, with no instructions file and no MCP servers. Each session could run only its tool and jq, and read and write files. It had no web tools and a spending limit of $3. The sessions used Claude Opus 4.8, the default model in Claude Code 2.1.216. pobear could escalate to the Web Data API, and Firecrawl ran without an API key.
• WebFetch baseline. Claude Code's WebFetch tool read the same 4 pages 3 times, at least 15 minutes apart, because it cached each URL for 15 minutes. WebFetch returns a model's answer to a prompt about the page and has no exit code, so the test checked its answers against the same expected values, with 1 fixed prompt per page.
• A page that loads its data from an API. Every tool ran 3 times on the /scroll page from the agent test, which has no quotes in its HTML and loads them from a JSON API.
• Datacenter IPs. The 4 local tools (pobear with --proxy and no Web Data API, crwl, Trafilatura, and curl + htmlq) reran through our datacenter proxy IPs in Germany, with a different IP for each run. Cloud agents and CI runners often send requests from datacenter IPs. These proxy IPs are similar to that setup, but they don't come from a cloud provider's own IP ranges.

The test measures how each tool behaves, so the results show what an agent sees on each page, not a success rate at scale. Every page is public and needs no login, and each protected site got 1 URL, tested at low volume over 2 days. Block rates change with request volume, time, IP reputation, and request headers, so a direct fetch that works for 3 requests can fail at 3,000.

Browser-automation CLIs for agents, such as agent-browser and Playwright CLI, click, type, and log in for the agent. Browser automation is a different job from fetching data, so this test focuses on fetching tools.

Results at a glance

This table compares the install, account, docs, and license of each tool:

This table compares the output, rendering, exit codes, and block behavior of each tool:

When the 3 runs gave different results, the cell lists each result. "Title only" means the Amazon page arrived without a US price, because Amazon showed the product as unavailable to our home connection. The Web Data API returned the price for the same product in the escalated runs and in a separate run with --provider decodo --template amazon. For pobear, the tier field shows the method that fetched the page, such as local (direct), proxy (through --proxy), or provider (the Web Data API). This table lists what each tool returned:

The free local tools differ in how they fail. pobear exited 5 every time a page had no data, on the 4 test pages, on the /scroll page, and through the datacenter IPs. crwl, Trafilatura, and curl + htmlq each exited 0 with no data on at least 1 page.

How the tools behave in agent setups

These 4 tests cover a new agent session, the agent's built-in fetch tool, a page that loads its data from an API, and requests from datacenter IPs.

An agent's first try

All 9 Claude Code sessions, 3 per tool, saved the 10 quotes with their authors on the first try:

Agent turns count the model's responses in each session, as Claude Code reported them. The pobear sessions found the page's free JSON API without instructions, so they needed no browser or hosted service. Firecrawl needed the fewest turns, because 1 hosted command rendered the page and extracted the fields. Most of each session's cost was the agent's own model usage. You pay this model cost once, when the agent writes the scraper, and the saved script then runs without a model.

Claude Code's built-in WebFetch tool

WebFetch read the static page correctly. On the other 3 pages, it missed some or all of the requested data, with the same results in all 3 runs:

WebFetch returned a model's answer instead of the page, so a missing price or a 403 arrived as text with no exit code to branch on. WebFetch works well for reading a page, but a CLI's output and exit code are easier to check for fields an agent must trust.

A page that loads its data from an API

Tools that read only the HTML got no quotes on /scroll, because its quotes arrive after the first response. The 3 runs per tool returned these results:

Fetching the page's JSON API needed no browser. With default settings, crwl's browser returned the page before the quotes loaded, so crwl exited 0 with no quotes.

Requests from datacenter IPs

Indeed returned no listings to any of the 4 local tools, and only pobear exited non-zero every time a page returned no data. The 3 runs per tool gave these results:

Indeed responded to curl, Trafilatura, and pobear with 403. crwl returned an empty array, although it got the listings from the home connection. Amazon served no CAPTCHA to these IPs, but it showed amazon.com with delivery set to Germany, where the product was unavailable and had no price.

When the data looks correct but is wrong

In 3 cases, a wrong or incomplete value arrived as a successful result. Firecrawl served a 21-hour-old cached Amazon page by default, Amazon showed no US price to connections in non-US countries, and WebFetch returned a different product name, taken from the title tag instead of the product heading. Before an agent uses a value, check its age, the location that the request was sent from, and the HTML element that it came from. pobear's --geo flag sets the location of a Web Data API request, and a schema check catches an empty field.

The 6 tools we tested

Each tool section covers what the tool is, how the test ran it, what it returned, and when to use it.

pobear: Free local scraping that escalates when blocked

What it is. pobear is our open-source web scraping CLI, a single Go binary under the MIT license. It fetches pages directly, extracts fields with CSS selectors (--css name=selector[@attr]) or converts HTML to Markdown (--extract markdown_), and crawls with depth and page limits. By default, a crawl stays on the same host and follows robots.txt. Each fetched page becomes 1 NDJSON object on stdout, with the extracted fields in data and the page's HTML in body, and the run summary goes to stderr. Give the agent only data, and keep body for checking selectors. Direct requests use Chrome, Firefox, or Safari TLS fingerprints, which you choose with --profile_. A TLS fingerprint is the pattern of settings that a client offers when it opens an encrypted connection.

pobear sends a page to a provider, which is a hosted scraping service, when the page needs more than a direct request. --ladder does this for blocked and rate-limited pages, and --render does it for pages that need JavaScript. The built-in provider is our Web Data API, which you select with --provider decodo. The same CSS selectors run on whatever it returns.

Agent features. These features let an agent learn and reuse pobear without a prompt that you write and maintain:

• Machine-readable docs. pobear docs --json prints the commands, the output contract, and the exit codes as JSON.
• Provider schema. pobear providers decodo --json lists the Decodo provider's templates and options, so the agent can find amazon or google without guessing.
• Reusable jobs. --print-spec prints a set of flags as a job spec, and pobear run reruns the saved spec.
• Pluggable providers. The Web Data API is built in. Providers are YAML manifests, so --ladder can also escalate to other scraping APIs that you configure.

Install. On Linux and macOS, run the install script:

curl -fsSL https://raw.githubusercontent.com/Decodo/pobear/main/install.sh | sh

The script checks the release checksum and installs to ~/.local/bin. Set POBEAR_VERSION to pin a release in agent setups. On Windows, download the ZIP for your system from the pobear releases page.

Test commands. These commands fetched each page directly:

pobear scrape https://books.toscrape.com/ --css 'title=h3 a@title' --css 'price=.price_color' --fail-on-empty
pobear scrape https://quotes.toscrape.com/js/ --css 'quote=.quote .text' --fail-on-empty
pobear scrape https://www.amazon.com/dp/B09B8V1LZ3 --css 'title=#productTitle' --css 'price=#corePrice_feature_div .a-offscreen' --fail-on-empty
pobear scrape 'https://www.indeed.com/jobs?q=python' --css 'title=.jobTitle' --fail-on-empty

For the --ladder row in the results table, the test added --ladder --provider decodo to these commands. For the Amazon page, the test also added --template amazon, because the Web Data API handles Amazon pages with a dedicated Amazon template. The same CSS selectors ran on the HTML that it returned.

Results. pobear returned all 40 fields on the static page with a direct request. On the JavaScript page, --render returned all 10 quotes through the Web Data API. You enable rendering per command, so you pay for it only where a page needs it.

On Indeed, --ladder escalated to the Web Data API and returned 43 to 44 job titles per run, with "tier":"provider" in the record. Indeed responded to the direct requests with 403, and pobear exited 5 with the URL marked [blocked].

Amazon responded to 2 of 3 direct requests with its CAPTCHA page and HTTP 200. pobear checks response bodies for the challenge markers of common anti-bot systems, including those of Amazon, Cloudflare, DataDome, and PerimeterX. In this test, that check marked Amazon's CAPTCHA pages [blocked], and --ladder escalated them the same way as a 403. For any other challenge page, --fail-on-empty makes the run exit 5 when no field matches. In the third direct run, Amazon served the product page without a US price. Nothing was blocked, so pobear exited 0 with the title only, and a schema check caught the missing price.

A docs site needs one crawl command. The command below returned 20 pages as Markdown records in under 4 seconds; we ran this crawl twice:

pobear crawl https://go.dev/doc/ --depth 1 --max-pages 20 --include 'https://go.dev/doc/**' --extract markdown --out docs.ndjson

When to use it. pobear is built for scraping jobs that run with no model reading the output. In those jobs, --fail-on-empty makes pobear exit non-zero when a run extracts nothing, and you pay only for the pages that need a provider. Pair it with a schema check, which catches a page with some fields missing.

Decodo CLI: Hosted rendering and ready-made scrapers

What it is. The Decodo CLI is the official command-line client for our Web Data API, written in TypeScript under the MIT license. Its main commands are scrape (Markdown by default), search, map, crawl, screenshot, and targets. Each API target, a ready-made scraper for 1 site or AI tool, also has its own command, such as google-search, amazon, walmart, reddit-post, and chatgpt. The decodo targets command lists the current set.

Install. Install it with npm install -g @decodo/cli on Node.js 22 or later, and authenticate it with a Web Data API key from the free plan. Save the key with decodo setup or set DECODO_AUTH_TOKEN. In a cloud agent with a network allowlist, add scraper-api.decodo.com and data.decodo.com.

Output. Output is Markdown by default, 1 JSON object per line with --format ndjson, and compact JSON when stdout is piped. The Decodo CLI README documents these exit codes:

Test commands. The test used these commands:

decodo scrape https://books.toscrape.com/
decodo universal https://quotes.toscrape.com/js/ --headless html --markdown
decodo amazon https://www.amazon.com/dp/B09B8V1LZ3
decodo scrape 'https://www.indeed.com/jobs?q=python'

decodo scrape fetches the page as the server sends it, and decodo universal --headless html renders it.

Results. The Decodo CLI returned the data on all 4 test pages with exit 0. The Amazon command returned parsed JSON instead of HTML, with fields such as price, stock, rating, and buy box under results. The Amazon scraper page lists the data points that the target covers. A shortened version of the output looks like this:

{"results": {"title": "Amazon Echo Dot (newest model) - Vibrant sounding speaker, Designed for Alexa+, Great for bedrooms, dining rooms and offices, Charcoal",
 "price": 79.99, "currency": "USD", "rating": 4.7, "reviews_count": 201302, "stock": "In Stock", "asin": "B09B8V1LZ3"}}

When to use it. The Decodo CLI runs every request through the Web Data API, which handles rendering, unblocking, and parsing, so it's the right choice when an agent needs parsed search results, eCommerce data, or AI answers from ChatGPT and Google AI Mode. We also ran decodo chatgpt and decodo google-ai-mode, and each returned JSON with the answer text, its citations, and links.

Firecrawl CLI: Hosted scraping with a keyless free tier

What it is. The Firecrawl CLI is the firecrawl-cli npm package, a Node.js client for Firecrawl's hosted API. Its commands include scrape (Markdown by default), search, map, crawl, interact, agent, and monitor.

Install. Its installer adds Firecrawl's agent skills to the coding agents it detects:

npx -y firecrawl-cli@latest init --all --browser

Account. Firecrawl's CLI docs describe a keyless free tier for some commands, rate-limited per IP. A free key raises the rate limits, and --api-url sets the CLI to use a self-hosted instance.

Test commands. The test used the default command with --json, which wraps the Markdown and the page metadata in 1 JSON object:

firecrawl scrape https://quotes.toscrape.com/js/ --json

Rendering happens in Firecrawl's cloud, and --wait-for adds a wait. Field extraction uses --format json --schema '…', which worked on the keyless tier in this test.

Results. The Firecrawl CLI returned the data on all 4 test pages with exit 0. For agents, 2 details matter:

• Cached copies by default. With default settings, all 12 runs returned cached copies, and the Amazon copy was about 21 hours old. Firecrawl's docs set a default maxAge of 2 days. A rerun with --max-age 0 fetched every page live with the same results, so use that flag for prices and anything else that changes.
• Exit 0 on failures. The docs list no exit codes, and the CLI exited 0 on a DNS failure, an HTTP 403, and an HTTP 500. A 403 or 500 appeared in metadata.statusCode, and a DNS failure returned "success": false with no metadata.

When to use it. Use Firecrawl when an agent writes a one-time scraper with you, because it needed the fewest turns in the agent test. In scripts, check success and metadata.statusCode instead of the exit code.

Crawl4AI CLI (crwl): Local JavaScript rendering for free

What it is. crwl is the command-line tool that is included in the Crawl4AI Python library (Crawl4AI CLI docs). It controls a local headless browser through Playwright, so it renders JavaScript without an account. Output can be markdown, markdown-fit, json (with an extraction schema), or all.

Install. Install the package, download the browser, and check the setup:

pip install crawl4ai
crawl4ai-setup
crawl4ai-doctor

For more detail, see our guide to setting up Crawl4AI step by step.

Test command. CSS extraction needs 2 files. The strategy file, extract_css.yml, selects CSS extraction:

type: "json-css"
params:
  verbose: false

The schema, quotes_schema.json, names the fields:

{"name": "Quotes", "baseSelector": ".quote", "fields": [{"name": "quote", "selector": ".text", "type": "text"}]}

The command passes both files:

crwl https://quotes.toscrape.com/js/ -e extract_css.yml -s quotes_schema.json -o json

Results. crwl returned all 40 fields on the static page and all 10 quotes on the JavaScript page. Its local browser returned 42 to 44 Indeed job titles in all 3 runs from the home connection, where Indeed responded to plain HTTP requests with 403. On Amazon from the home connection, it received the CAPTCHA page in every run. It exited 0 on an HTTP 403 page and exited 1 on a DNS failure.

When to use it. crwl renders JavaScript in a local browser for free. On pages that fetch their data after loading, such as /scroll, crwl needs a wait_for condition. It needs Python and a browser download, and it starts a browser on every run, which adds time. Its docs list no exit codes.

The -b proxy=<url> option routes crwl's browser through a proxy, and the datacenter test used that option. For residential IPs, use our residential proxies with the same option. The library has its own anti-bot fallback options, and the CLI has no flag to escalate to a hosted scraping API. Crawl4AI also offers a hosted cloud service with its own credits.

Trafilatura: Clean main-text extraction

What it is. Trafilatura is a Python package and CLI that extracts the main text and metadata from articles (trafilatura CLI docs). Output is TXT by default, or --markdown, --json, --csv, --xml, and --html.

Install. Install it with pip install trafilatura.

Test command. Trafilatura extracts main text rather than fields matched by CSS selectors, so the test requested JSON and checked whether titles and prices appeared in the text:

trafilatura -u "https://books.toscrape.com/" --json

Results. Trafilatura returned 20 of 40 fields on the static page and no quotes on the JavaScript page, with exit 0 on both. On the static page, each link shows a shortened title, and the full title is only in the link's title attribute. Trafilatura extracts text, not attributes, so the output contained the prices but no full titles. On Indeed's 403 response, it exited 126, a code its docs don't list. On Amazon's CAPTCHA page, it exited 0 and returned the "continue shopping" text as the article.

When to use it. Of the 6 tools, it's the one built for article text and corpora. It reads the HTML as the server sends it, without rendering JavaScript, and it works best on article pages rather than listing pages. For a Python approach, read our guide to extracting text from HTML with Python.

curl and htmlq: the zero-dependency baseline

What it is. curl fetches the page, and htmlq applies CSS selectors to it (--text, --attribute).

Install. curl is installed on most systems. Install htmlq with brew install htmlq or cargo install htmlq.

Test commands. The static page needs 1 command per field:

curl -fsS https://books.toscrape.com/ | htmlq --attribute title 'h3 a'
curl -fsS https://books.toscrape.com/ | htmlq --text '.price_color'

Results. curl + htmlq returned all 40 fields on the static page, no quotes on the JavaScript page, Amazon's product title without a price, and nothing on Indeed, with exit 0 on every page.

Exit behavior. -f makes curl exit non-zero on HTTP errors. curl returned code 22 on Indeed's 403 over HTTP/1.1, and code 56 when the 403 arrived over HTTP/2. In a pipeline, though, the shell returns htmlq's exit code, which was 0 with empty output.

Add set -o pipefail to any script an agent runs, so the pipeline returns curl's failure code. On a challenge page with HTTP 200, the pipeline exits 0 with empty output, with or without pipefail, so the agent needs to check for empty output itself.

When to use it. There's no account to create and almost nothing to install. When a job also needs crawling, JavaScript rendering, or escalation, pobear accepts the same CSS selectors in 1 binary and adds --fail-on-empty and --ladder. pup, an older HTML selector tool, does the same job, but its last release was in 2016, so htmlq is the one to use with curl. For more on fetching with curl, see scraping with curl, step by step.

How to run pobear from an AI agent

The workflow below keeps every fetch free until a page needs the Web Data API. Before an agent scrapes a site, check the site's terms of service and robots.txt, and collect personal data only when you have a legal basis for it. Our guide to whether web scraping is legal covers the basics.

1. Install pobear. Use the install script from the pobear section, or the release ZIP on Windows. The script in step 5 also uses jq and check-jsonschema. Install jq with brew install jq or your system's package manager, and install check-jsonschema with uv tool install check-jsonschema or pipx install check-jsonschema.

2. Add a Web Data API key. Direct fetching runs without a key, and any command with --provider decodo reads the key you save once. Copy your key from the Decodo dashboard, run pobear auth decodo, and paste the key at the decodo token prompt. In an agent sandbox or CI, where nobody can answer a prompt, set POBEAR_PROVIDER_DECODO_TOKEN instead. In a cloud agent with a network allowlist, add data.decodo.com, the host that pobear's provider calls. The Web Data API pricing page lists the free plan and the paid plans.

In cloud agents, the sandbox's network rules can block a direct fetch before the site's bot check runs. Allowlisting 1 API host is simpler than allowlisting every site the agent visits. In a sandbox with a network allowlist, run pobear with --provider decodo and no --ladder, which sends every request to the API. Where direct access is open, keep --ladder, so pages that block datacenter IPs still reach the Web Data API. In this test, Indeed blocked every local tool from the datacenter IPs.

To add a proxy between direct fetching and the API, pass it to --proxy. --ladder then tries a direct request, then the proxy, then the Web Data API, and it stops at the first tier that isn't blocked. --ladder treats a 403, a detected challenge page, or a rate limit as blocked.

In our test with one of our datacenter proxies, the command below tried all 3 tiers on Indeed and returned the job titles from the Web Data API:

pobear scrape 'https://www.indeed.com/jobs?q=python' --css 'title=.jobTitle' --ladder \
  --proxy 'http://YOUR_PROXY_USERNAME:YOUR_PROXY_PASSWORD@dc.decodo.com:10001' --provider decodo

On Amazon, the same ladder stopped at the proxy because the datacenter IP got the product page. For residential proxies, pass the gate.decodo.com gateway to the same flag. In a cloud agent with a network allowlist, add the proxy host too.

3. Let the agent learn the tool. Part of the pobear docs --json output looks like this:

{
  "exitCodes": [
    {"code": 0, "meaning": "everything succeeded"},
    {"code": 1, "meaning": "auth failure: a credential is missing or was rejected"},
    {"code": 3, "meaning": "usage error or invalid job spec, nothing was fetched"},
    {"code": 4, "meaning": "some items failed; the ones that succeeded are on stdout"},
    {"code": 5, "meaning": "every item failed, or the failure is unclassified"}
  ],
  "output": {
    "stdout": "one NDJSON object per fetched item",
    "itemFields": ["url", "finalUrl", "parentUrl", "status", "body", "depth", "data", "tier"]
  }
}

In the agent test, every pobear session ran pobear docs on its own, with no instructions file. If you add a line to AGENTS.md or CLAUDE.md, add only 1 rule about the output:

Read only the data field of each pobear record.

4. Find the data before writing selectors. Selectors can break when the markup changes, so first check whether the site already publishes the data in a stable form:

• Markdown and structured data. Some sites return their own Markdown when a request sends Accept: text/markdown. With -H 'Accept: text/markdown', pobear got a 7 KB Markdown version of the Workers page from Cloudflare's developer docs, about a fifth of the 36 KB that --extract markdown converted from its HTML. Many pages also include JSON-LD blocks (script[type="application/ld+json"]).
• Embedded JSON and JSON APIs. Many pages include their data as JSON in a script tag, as the JavaScript page does, or load it from a JSON API that the page calls, as the /scroll page does. The browser's network tab shows that API. On the /scroll page, 1 free request returned the data:

pobear scrape 'https://quotes.toscrape.com/api/quotes?page=1' --bare \
  | jq -c '.quotes[] | {quote: .text, author: .author.name}'

That pipeline returned all 10 quotes with their authors in about 1 second. Run it with set -o pipefail, so a failed fetch makes the whole pipeline exit non-zero.

When the site offers none of these, write CSS selectors and run them with the script in step 5.

5. Escalate only when needed. --ladder escalates hard blocks and the challenge pages that pobear detects. The script below also handles empty and incomplete results, and validates every record against a JSON Schema. The script uses the JavaScript page as its example of a page that needs rendering. Save the script as scrape.sh:

#!/usr/bin/env bash
set -o pipefail
URL="${1:-https://quotes.toscrape.com/js/}"
FIELDS=(--css 'quote=.quote .text' --css 'author=.quote .author')
SCHEMA="$(dirname "$0")/quotes.schema.json"

# Try direct first, and --ladder escalates to Decodo on blocks and rate limits
pobear scrape "$URL" "${FIELDS[@]}" --ladder --provider decodo --fail-on-empty --out quotes.ndjson
status=$?

# Exit 5 with a record in the file means the page loaded but nothing was extracted
if [ "$status" -eq 5 ] && [ -s quotes.ndjson ]; then
  pobear scrape "$URL" "${FIELDS[@]}" --provider decodo --render --fail-on-empty --out quotes.ndjson
  status=$?
  # Still empty after rendering, so check the selectors before paying for another retry
  if [ "$status" -eq 5 ] && [ -s quotes.ndjson ]; then
    echo "scrape.sh: rendered page also empty, check the selectors" >&2
    exit 10
  fi
fi

# Exit 0 means at least 1 field matched, so validate every record against the schema
if [ "$status" -eq 0 ]; then
  jq -s 'map(del(.body))' quotes.ndjson > quotes.json
  if ! check-jsonschema --schemafile "$SCHEMA" quotes.json >&2; then
    exit 11
  fi
fi

exit "$status"

On this page, the direct fetch succeeds with HTTP 200 and 0 quotes. --fail-on-empty changes the exit code to 5, and because pobear still writes the record, the script retries through the Web Data API with --render. The second run overwrites the same file, so the agent reads only the second attempt. In a sandbox with a network allowlist, remove --ladder from the script's first command.

The script adds 2 exit codes of its own, outside the ranges that pobear and the Decodo CLI use:

• Exit 10: the rendered page is also empty. The site may have changed its markup or served a different page. The record stays in quotes.ndjson, so check its body and fix the selectors before the next rendered request.
• Exit 11: a record failed the schema. --fail-on-empty catches pages with no fields, and the schema check catches a page with quotes but no authors. check-jsonschema names the failing field on stderr, such as $[0].data.author: [] should be non-empty, so check that field's selector first. If the page itself arrived incomplete, retry through the provider.

The schema, saved as quotes.schema.json next to the script, requires a URL and at least 1 non-empty quote and author in every record:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "array",
  "minItems": 1,
  "items": {
    "type": "object",
    "required": ["url", "data"],
    "properties": {"data": {"type": "object", "required": ["quote", "author"],
      "properties": {"quote": {"$ref": "#/$defs/texts"}, "author": {"$ref": "#/$defs/texts"}}}}
  },
  "$defs": {"texts": {"type": "array", "minItems": 1, "items": {"type": "string", "minLength": 1}}}
}

We tested the script on 5 outcomes, and each exited with the code described above. The outcomes were a rendered retry, a wrong selector, a missing field, a network error, and a direct fetch that succeeded on a page that needs no JavaScript. A network error exited 5 without a paid retry.

6. Run it. The agent runs the script and reads the exit code:

bash scrape.sh; echo $?

7. Read the output. The script prints pobear's summaries and the schema result on stderr, and echo prints the exit code. On the JavaScript page, the first 2 lines are the direct attempt, which found no quotes; the third is the rendered retry, and the fourth is the schema check:

fetched 1, errors 0, skipped 0, empty 1
pobear: every fetched item extracted nothing (1 items)
fetched 1, errors 0, skipped 0
ok -- validation done
0

The record in quotes.ndjson looks like this, with the HTML body removed and the quote lists shortened:

{"url":"https://quotes.toscrape.com/js/","finalUrl":"https://quotes.toscrape.com/js/","parentUrl":"","status":200,"depth":0,"data":{"author":["Albert Einstein","…"],"quote":["“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”","…"]},"tier":"provider"}

The tier field shows which method fetched the page; here, provider for the Web Data API. To give the agent only the extracted fields, read data:

jq -c '.data' quotes.ndjson

For the JavaScript page, jq reduces the 11 KB file to about 1 KB of quotes and authors. For an escalated Amazon page, the extracted fields are 185 bytes.

Exit codes map to these actions:

The table skips exit code 2, which pobear reserves for Go's runtime. The same table works outside agents, for example in CI/CD pipelines that should save only runs with data.

Run the job on a schedule

A recurring job needs 3 practices that a single run doesn't need:

• Use --fail-on-empty on every run. Schedule pobear scrape or scrape.sh, so an empty page exits 5.
• Write each run to its own file. The last good result then stays unchanged after a failed run. scrape.sh writes quotes.ndjson in the working directory, so start each run in a new folder, such as a folder named with the run's timestamp.
• Route by site. For a site that you already know needs the Web Data API, use --provider decodo without --ladder. Each blocked direct attempt costs time and is a request from your own IP that the site can mark as suspicious.

For Amazon, this command returned the title and the US price that direct requests from elsewhere didn't get:

pobear scrape https://www.amazon.com/dp/B09B8V1LZ3 --css 'title=#productTitle' \
  --css 'price=#corePrice_feature_div .a-offscreen' --provider decodo --template amazon --fail-on-empty

Reproduce the tests

The tool sections show each tool's test command, and the test-page table lists the CSS selector for every field. The agent test, the WebFetch baseline, and the datacenter test need the details below.

Agent test. Each session ran from an empty folder with this command, shown here for pobear:

claude -p "Use the pobear command-line tool, which is installed, to get all 10 quotes and their authors from https://quotes.toscrape.com/scroll. Save them to quotes.json in the current folder as a JSON array of objects with \"quote\" and \"author\" keys. Use pobear for every web request. Don't write your own HTTP code or use any other scraping tool." \
  --setting-sources project --strict-mcp-config --mcp-config '{"mcpServers":{}}' \
  --allowedTools 'Bash(pobear *)' 'Bash(pobear)' 'Bash(jq *)' Write Read \
  --disallowedTools WebFetch WebSearch --no-session-persistence --max-budget-usd 3 \
  --output-format stream-json --verbose

The crwl and Firecrawl sessions used the same command, with crwl or firecrawl in place of pobear in the prompt and in --allowedTools.

WebFetch baseline. WebFetch got 1 fixed prompt per page:

Static page:     List every book title and its price on this page exactly as written, one per line, in the form: title | price. If there are none, say NONE.
JavaScript page: List every quote text on this page exactly as written, one per line. If there are none, say NONE.
Amazon:          Return the exact product title and the exact price as shown on this page, one per line, in the form: title: ... and price: ... If the page is a CAPTCHA or robot check, say CAPTCHA.
Indeed:          List every job title on this page exactly as written, one per line. If there are none, say NONE.

Datacenter test. pobear used --proxy, crwl used -b proxy=<url>, curl used -x, and Trafilatura read a SOCKS5 proxy URL from the http_proxy environment variable. Trafilatura needs the PySocks package for a SOCKS5 proxy.

Which web scraping CLI should your agent start with?

Start with pobear for unattended jobs: it was the only local tool that exited non-zero on every empty or blocked page, and --ladder sent each blocked page – 2 on Amazon, 3 on Indeed – to the Web Data API, which returned the data. Choose the Decodo CLI for jobs where you want parsed results instead of selectors, such as search results, eCommerce data, and AI answers. Among the other tools, Firecrawl reached a correct result in the fewest agent turns, and crwl is the free choice for local JavaScript rendering.

For live web access for AI agents on pages that block direct requests or need JavaScript, get a Web Data API key on the free plan. Then add --ladder --provider decodo to your pobear command, and --render where a page needs JavaScript.