Cloudflare error 520 means Cloudflare reached the origin server but got an empty, unknown, or malformed response back. For scrapers, the important clue is context. If the page loads in a normal browser but returns 520 from your script, the origin is probably killing your automated connection because the request looks automated. You'll learn what 520 means, why scrapers trigger it, how to fix it from the client side, and how it differs from nearby Cloudflare 5xx errors.

TL;DR

• Cloudflare returns error 520 when it reaches the origin, but receives no valid HTTP response
• Scrapers often trigger 520 when bot logic kills a suspicious connection instead of returning 403 or a challenge
• Log the Ray ID, retry once after a short delay, then use exponential backoff before changing scraper settings
• Fix persistent 520s by aligning headers, TLS fingerprints, cookies, concurrency, and IP reputation

What is Cloudflare error 520?

Cloudflare error 520 is Cloudflare's catch-all 520 status code for "Web server is returning an unknown error." It appears when the origin server returns an empty, malformed, or unexpected response.

The origin server is the target site's backend server behind Cloudflare's proxy. Cloudflare sits in front of it as a reverse proxy, which means your scraper talks to Cloudflare first, then Cloudflare forwards the request to the origin.

When the origin sends a response that Cloudflare can't parse as valid HTTP, Cloudflare generates the 520 page at the edge server handling your request.

Cloudflare error 520 is Cloudflare-specific

520 isn't a standard HTTP status code, but a Cloudflare-specific one. You see it only when the target sits behind Cloudflare's proxy.

A site that doesn't use Cloudflare usually exposes the underlying failure directly, such as a 500, a 502, a connection reset, or a timeout.

How Cloudflare error 520 appears

For scraping, 520 often looks stranger than it is: your browser works, but your scraper fails. The same URL behaves differently depending on headers, TLS handshake, cookies, IP range, and request timing.

Many protected origins terminate suspicious automated connections without returning a clean 403 or CAPTCHA. Cloudflare can't reliably tell that from an origin crash, so it wraps the failed exchange as 520.

How Cloudflare 520 error appears

When you see the Cloudflare error page, capture the Ray ID. A Ray ID is Cloudflare's identifier for a specific request at its edge.

It usually appears at the bottom of the error page and may also appear in the cf-ray response header. Log it with the URL, timestamp, proxy IP, request headers, and retry result.

Why your scraper triggers Cloudflare error 520

Most scraper-side 520 errors come from signals your client controls: IP reputation, browser fingerprints, headers, cookies, and request volume. The tricky part is that Cloudflare reports all of these as an "unknown error" because the origin closes or breaks the response before Cloudflare can classify it cleanly.

Use the causes below to narrow down whether you're seeing a bot block, malformed request, overload issue, or temporary origin failure.

Bot detection terminates your scraper connection

The most common scraper-side cause is silent bot termination. The origin's bot layer flags your request based on signals like TLS fingerprint, header order, missing browser hints, suspicious cookies, or IP reputation, then closes the connection without writing a normal response. Cloudflare sees a broken response and returns 520.

The tell: real browser traffic from the same network works, but automated traffic fails.

That behavior sits in the broader world of anti-bot systems. A strict site doesn't need to send a neat "blocked" page. It can drop the connection, return malformed output, or let an upstream component die before response headers are complete. Those paths all look like origin weirdness from Cloudflare's side.

Bad IP reputation on datacenter proxies

If you're scraping from a cloud VM, a noisy datacenter proxy pool, or an ASN associated with abuse, the origin may distrust the whole range.

An ASN identifies the network that announces a block of IPs on the Internet. When 520s correlate with IP range rather than URL, you're probably dealing with reputation, not broken selectors. Using residential proxies here can help you with the request reputation.

TLS and HTTP fingerprints don't match your User-Agent

TLS and HTTP fingerprints can contradict the browser identity your scraper claims. A TLS fingerprint, such as JA3 or JA4, summarizes details from the TLS handshake. If Python requests and urllib3 produce a non-browser fingerprint while your User-Agent claims Chrome, a strict origin can spot the mismatch. The same goes for HTTP/2 behavior that doesn't match the browser you're claiming.

Browser headers don't match a real request

A real Chrome request normally includes a coherent set of Accept, Accept-Language, Accept-Encoding, sec-ch-ua, and sec-ch-ua-platform headers. If your scraper sends a Chrome User-Agent with missing client hints, odd header ordering, or values no real browser would send, the origin has an easy reason to distrust it.

Oversized cookies and headers trigger 520

Cloudflare currently lists headers exceeding 128 KB as a common cause of error 520, often because of excessive cookies. Scrapers create this problem easily. A long-running stateful client keeps accumulating cookies across redirects, sessions, and retries until the request becomes much larger than a normal browser session.

High request volume can break the origin

If parallel requests exhaust application workers, memory, or database connections, Cloudflare may establish the connection but still receive no valid response because the origin dies mid-request.

In that case, 520s usually arrive in bursts that line up with concurrency spikes.

Transient origin issues cause one-off 520 errors

Some 520s are just transient origin noise. A deploy, worker crash, overloaded upstream, or temporary Cloudflare-to-origin glitch can produce 1 failed request, then disappear. Treat those as retryable, but don't let them hide a pattern. A random 520 and a repeatable scraper-only 520 are different problems.

How to fix or avoid Cloudflare error 520 in your scraper

Start with the lowest-effort checks, then move toward fingerprint, traffic, and proxy fixes if 520s keep repeating.

Fix 1: Log the Ray ID and retry once

Capture the Ray ID from the Cloudflare page or the cf-ray header, then retry after 2–5 seconds. If the retry succeeds, you probably hit a transient origin problem. Log it, count it, and move on.

Fix 2: Use exponential backoff with jitter

Production scrapers need retry discipline. For 520-prone requests, use exponential backoff with jitter:

wait = min(base * 2 ** attempt + random(0, 1), max_wait)

Cap retries at 3–5 attempts. Immediate retries can worsen a stressed origin and make bot patterns louder.

Fix 3: Send a complete, browser-realistic request

Send headers that match the browser identity you're claiming. Use a real recent User-Agent, then align Accept, Accept-Language, Accept-Encoding, sec-ch-ua, sec-ch-ua-platform, and a plausible Referer where the request flow calls for it. If you're using HTTP/2, keep header behavior close to what the browser actually sends.

Fix 4: Align your TLS fingerprint with your User-Agent

A Chrome User-Agent doesn't help if the TLS handshake says Python requests. Use a TLS-fingerprint-aware client such as curl_cffi with browser impersonation, or use a real Chromium-based headless browser when the target requires it. The goal is to make the network, HTTP, and browser layers tell the same story.

Decodo's browser fingerprinting guide gives more background on why these signals matter.

Fix 5: Trim your cookie chain

Audit the size of your Cookie header across long sessions. If it grows into tens of KB, clear non-essential cookies, isolate sessions per target, and rotate sessions sooner.

Don't reuse a single cookie jar across unrelated URLs just because your HTTP client makes it easy. Cloudflare currently calls out headers over 128 KB, but you should treat rapid cookie growth as a warning long before it reaches that ceiling.

Fix 6: Cap concurrency and throttle per host

If 520s appear during request spikes, reduce pressure before changing fingerprints. Start with 5–10 simultaneous requests per host and add 1–3 seconds between requests on the same connection. Then raise concurrency slowly while watching the 520 rate. Throttling also reduces the repetitive timing pattern that makes automated traffic easier to classify.

Fix 7: Rotate through residential IPs when ASN reputation is the trigger

If the same scraper works from your home connection but fails from a datacenter range, the IP layer is the likely issue. Rotating proxies help spread requests across different IPs, but the proxy type matters. Decodo residential proxies route traffic through an ethically-sourced residential IP network, helping you avoid failures tied to noisy datacenter ranges.

Fix 8: Use a managed unblocker or scraping API for the hardest targets

Some sites layer custom bot logic on top of Cloudflare specifically to break manual automation. If you still get 520 after header fixes, TLS alignment, residential rotation, and throttling, stop tuning the same script forever.

Decodo Site Unblocker handles proxy rotation, JavaScript rendering, browser fingerprinting, cookies, headers, retries, and fallbacks through a single endpoint.

If your pipeline needs structured data instead of raw HTML, Decodo Web Scraping API can return results in formats such as HTML, JSON, CSV, XHR, PNG, and Markdown while handling retry and rendering work upstream.

Cloudflare error 520 vs other Cloudflare 5xx errors

Cloudflare 5xx errors describe different failures between Cloudflare and the origin server. Knowing the difference keeps you from fixing the wrong layer.

For scrapers, use this rule: if Cloudflare reached the origin and got nonsense or nothing, it's 520. If Cloudflare couldn't connect or didn't get a timely acknowledgment, it's 521, 522, or 523. If the origin accepted the connection but took too long to send a response, it's 524. If TLS fails between Cloudflare and the origin, it's 525 or 526.

Don't confuse origin-relay errors with Cloudflare 1xxx edge blocks. Errors like 1015, 1020, 1009, and 1005 usually mean Cloudflare itself made an access-control decision at the edge. A normal 500 internal server error, by contrast, usually points to the origin application itself, even if Cloudflare obscures the exact failure.

Note for site owners

If you run the Cloudflare-protected site, confirm where the failure happens before changing firewall, proxy, or application settings. Reproduce the failing URL with curl and --resolve, pinned to your origin IP, so the request bypasses Cloudflare. If the origin fails directly, debug the origin. If the origin works directly but fails through Cloudflare, focus on the Cloudflare-to-origin path.

The 3 highest-leverage checks are:

1. Allowlist the current Cloudflare IP ranges in your origin firewall and refresh that list regularly.
2. Audit Set-Cookie chains and custom response headers so total headers stay below Cloudflare's current 128 KB limit.
3. Review application logs for crashes, out-of-memory kills, process restarts, or malformed responses that match the 520 timestamp and Ray ID.

Conclusion

Cloudflare error 520 on automated requests usually means the origin's bot layer flagged your request and killed the connection in a way Cloudflare could only label as "unknown." Start by logging the Ray ID, retrying once after a short delay, and using backoff only if the error repeats. Then fix the signal that actually triggered the failure: headers, TLS fingerprint, cookies, concurrency, or IP reputation. For teams that don't want to chase 520-class failures every week, Decodo residential proxies, Site Unblocker, and Web Scraping API help remove the trigger instead of patching the symptom.