One URL In, Full Site Out With a Web Crawler API

Q: What's the difference between /crawl, /scrape, and /map?

_/scrape_ fetches one URL and returns its content as HTML, Markdown, or XHR output, plus JSON, CSV, or PNG for specific targets. _/map_ returns the URLs it finds on a site, so you can see what's there before pulling anything. _/crawl_ combines the two. It discovers pages across a site and scrapes each one, up to 10K pages per job. In short, web crawling finds pages and web scraping extracts content from them. _**/map**_ crawls, _**/scrape**_ scrapes, and _**/crawl**_ does both in one job.

Q: How does /crawl find pages?

It uses the same discovery as _/map_. _/map_ returns a first-level sitemap plus the links it follows along the way. _/crawl_ builds its URL list from that, filters the list by the parameters you pass, and scrapes each remaining page. Results come back page by page as they're collected. For longer jobs, you can set a callback URL, and _/crawl_ notifies your endpoint when the run finishes.

Q: Can I crawl only part of a site?

Yes, Decodo's _/crawl_ endpoint filters the discovered URL list by your request parameters before scraping, so you only pay for and process the pages you need.

Q: Does /crawl work on JavaScript-heavy sites?

Yes. When a page needs JavaScript rendering, _/crawl_ retries it with rendering automatically, so you don't need separate logic for static and dynamic pages. This makes it a practical way to crawl a JavaScript website without running your own headless browsers.

Decodo's /crawl endpoint autonomously pulls up to 10K pages from a site, no link-following logic to write, no manual pagination. Advanced features, including 125M+ IPs, help to access even the most protected pages, and failed pages get an automatic retry with JS rendering.

Start for free Start free with Google

Try an example

Plugs directly with:

100+

ready-made templates

99.99%

success rate

200

requests per second

125M+

IPs worldwide

Up to 5K

free requests

Turn one URL into a full dataset

Building a crawler usually means writing your own queueing, retry, and rate-limiting logic. Decodo's /crawl endpoint handles that for you. It discovers URLs the same way /map does, filters them by the parameters you pass, and scrapes each page. You get the results back and process them however you need.

Build a RAG knowledge base in a single job

Point /crawl at a docs site or help center and get each page back as clean Markdown, ready to chunk and embed.

Give failed pages a second shot

Switch on JS retry and each failed page gets another attempt with JavaScript rendering. If it fails again, it's skipped, so a single bad URL doesn't stall the crawl.

Get notified instead of polling

Set a webhook callback and pick up results the moment the job finishes.

Choose the output your pipeline expects

Get each page as Markdown or HTML, same as /scrape, so a crawl slots into a RAG pipeline without an extra conversion step.

Scope a crawl with /map first

Run /map first to pull up to 147K URLs from a site and its sitemap, then crawl only the sections you need.

Reach sites that block DIY crawlers

Every page in a crawl runs through the same infrastructure as /scrape – 125M+ IPs, built-in proxy management, and a 99.99% success rate.

See the request and response shapes in the /crawl documentation

Read docs

Trusted by:

Pricing that scales with your web data needs

Start with a free plan and up to 10K recurring requests, then choose a subscription that best matches your use case and volume.

10 req/s

/month

Up to 5K crawls or 10K pages scraped

Start for free

10 req/s

$19

/month

Up to 19K crawls or 38K pages scraped

Buy now

25 req/s

$49

/month

Up to 65K crawls or 122K pages scraped

Buy now

50 req/s

$99

/month

Up to 141K crawls or 330K pages scraped

Buy now

100 req/s

$249

/month

Up to 383K crawls or 1.2M pages scraped

Buy now

Need more?

Chat with us and we’ll find the best solution for you

Contact sales

With each plan, you access:

JavaScript rendering

99.99% success rate

HTML, Markdown, XHR, plus JSON, CSV, PNG where supported

One-click AI integrations

LLM-ready Markdown format

100+ pre-built templates

Single API key for all endpoints

/scrape, /search, /map, /crawl

24/7 tech support

14-day money-back

SSL Secure Payment

Your information is protected by 256-bit SSL

Frequently asked questions

What can I actually build with the /crawl endpoint?

Anything that needs a full site's worth of content, not just one page – a RAG knowledge base, a documentation mirror, a large-scale content audit, or a dataset for training or analysis.

Is /crawl synchronous, or do I need to check back for results?

It runs as a background job. Poll for status, or set a webhook to get notified when it's done.

How long are results available after a crawl finishes?

Roughly 24 hours. Use a webhook to capture results as soon as the job completes if you need them longer.

What happens if a page fails to load?

One automatic retry with JS rendering. If that also fails, the page is skipped rather than blocking the rest of the crawl.

Is there a free plan?

Yes, you can get started with a free plan that's worth $5 or up to 10K requests, no credit card required. The plan renews each month, so you can run your small-scale projects with the free plan, and when you're ready, grab a subscription that suits your needs best.

What's the difference between /crawl, /scrape, and /map?

/scrape fetches one URL and returns its content as HTML, Markdown, or XHR output, plus JSON, CSV, or PNG for specific targets. /map returns the URLs it finds on a site, so you can see what's there before pulling anything. /crawl combines the two. It discovers pages across a site and scrapes each one, up to 10K pages per job.

In short, web crawling finds pages and web scraping extracts content from them. /map crawls, /scrape scrapes, and /crawl does both in one job.

How does /crawl find pages?

It uses the same discovery as /map. /map returns a first-level sitemap plus the links it follows along the way. /crawl builds its URL list from that, filters the list by the parameters you pass, and scrapes each remaining page. Results come back page by page as they're collected. For longer jobs, you can set a callback URL, and /crawl notifies your endpoint when the run finishes.

Can I crawl only part of a site?

Yes, Decodo's /crawl endpoint filters the discovered URL list by your request parameters before scraping, so you only pay for and process the pages you need.

Does /crawl work on JavaScript-heavy sites?

Yes. When a page needs JavaScript rendering, /crawl retries it with rendering automatically, so you don't need separate logic for static and dynamic pages. This makes it a practical way to crawl a JavaScript website without running your own headless browsers.

Am I charged for pages that fail?

No, Decodo’s Web Data API retries the same request until it returns the data you need, and you’re not charged for the pages that failed to fetch.

Stop Rebuilding Your Own Crawler

Point /crawl at a site and get a dataset back, not a queueing system to maintain. Pair it with /map to scope exactly what you pull. Up to 10K free recurring requests included, no credit card required.

Start for free