What is Charles Proxy: Traffic Inspection, Debugging, And Web Scraping Guide

TL;DR

• Charles Proxy works as a man-in-the-middle proxy that intercepts and decrypts HTTP/HTTPS traffic for debugging, testing, and scraping.
• The tool allows users to inspect, edit, replay, and rewrite requests and responses in real time, including headers, cookies, payloads, and API calls.
• Charles supports desktop and mobile traffic monitoring, SSL proxying, bandwidth throttling, WebSocket inspection, and network simulation across macOS, Windows, Linux, iOS, and Android.
• Web scrapers use Charles to discover hidden API endpoints, capture authentication headers and tokens, replicate browser requests, and troubleshoot anti-bot protections.
• While powerful for traffic analysis and development, Charles Proxy is best suited for debugging and workflow discovery rather than large-scale production scraping.

What is Charles Proxy and how does it work?

Charles Proxy is a HTTP proxy and cross-platform web debugging application that runs locally and intercepts all HTTP/HTTPS traffic passing through it. It works as a local proxy server that sits between a client and the Internet, often defined as a man-in-the-middle debugging proxy, because it captures HTTP/HTTPS traffic between applications/browsers and servers.

Without the proxy, your app communicates with the server directly, so the connection is encrypted and you can’t see what’s being sent. But with Charles, the device sends the traffic through it first, allowing you to see all the raw data in plain text.

More precisely, Charles Proxy routes browser or device traffic through a local proxy (with a default port 8888) and captures the request/response cycles in their entirety, including headers, cookies, payloads, and timing data. It also captures network and session metadata (i.e. hostnames, URLs, protocol, port), request/response size (i.e. the total size of data exchanged), client info, and specialized data types. Developers and testers can intercept, inspect, and modify this traffic in real time.

The process provides a glimpse into network communications for developers, testers, web scrapers, and anyone who needs to inspect requests. This is useful for debugging APIs, reverse-engineering apps, analyzing web behavior, improving performance, and simulating network conditions without changing the backend code. Charles Proxy can also capture website traffic, analyze dynamically loaded content, and help reverse-engineer APIs for more efficient scrapers.

The left-hand panel offers two traffic analysis views. Structure view organizes requests by domain and URL path in a tree, making it useful for navigating complex APIs and finding endpoints. Sequence view lists requests chronologically, helping debug traffic flow, identify latency, and track API call order.

Charles Proxy supports macOS (including Apple Silicon), Windows, and Linux, and can automatically configure proxy and browser settings on macOS and Windows. Because it is Java-based, it runs across multiple desktop operating systems and can also capture traffic from mobile devices on the same network.

Charles offers a fully featured 30-day free trial. After that, users can choose between a single-user license or a site license for multiple computers.

Key features and capabilities

With SSL proxying enabled through its root certificate, Charles can decrypt HTTPS traffic for analyzing modern web and mobile apps. Breakpoints let you pause and edit requests or responses, while the Rewrite tool automatically modifies headers, URLs, parameters, or responses without changing code.

Map Remote and Map Local allow request redirection to different servers or local files for easier frontend and backend testing. Charles also supports bandwidth throttling to simulate slow network conditions and includes native WebSocket inspection.

Captured sessions can be recorded, replayed, and exported for offline analysis, bug reproduction, or team sharing. It supports JSON, JSON-RPC, SOAP, and XML-RPC in a readable format.

Installing and configuring Charles Proxy

In this section, we’ll provide a simple, step-by-step guide to help you easily install Charles Proxy and configure it for different environments.

When launched, Charles can automatically configure your OS and browser. You can find this option in the browser menu, or go to the Proxy menu and check Windows Proxy or macOS Proxy.

Alternatively, you can set it up manually, typically like this:

1. Run Charles on your device.
2. Configure the proxy by setting your browser or mobile device to use your computer's IP address and port 8888.
3. Install the Charles Root Certificate on your device for HTTPS inspection.
4. Inspect and modify data.

Desktop setup

To set up Charles Proxy on desktop, download and install the application from the official website.

1. Open Charles and go to Proxy.
2. Charles will attempt to auto-configure system proxy settings on launch; check that web traffic appears in the main window in order to verify.
3. In Proxy Settings, enter the computer's Local IP address and configure the port (default proxy runs on localhost 8888).
4. Enable Proxy, Windows/macOS Proxy and install the Root Certificate via Help → SSL Proxying.

Mobile device configuration (iOS)

To configure Charles Proxy on iOS, connect your iOS device to the same Wi-Fi network running the proxy. 

• Open Charles, go to Help → Local IP Address to get your computer’s IP.
• Go to Settings → Wi-Fi → the info icon next to your network.
• Click Configure Proxy and set it to Manual.
• Set Server to your computer's IP and Port to 8888, then Save.
• A prompt will appear → allow connection.
• Open Safari on iOS and visit chls.pro/ssl to install the SSL certificate in Settings → General → VPN & Device Management.
• Trust the certificate in Settings → General → About → Certificate Trust Settings → enable Charles Proxy CA.
• In Charles, go to Proxy → SSL Proxying Settings → click Enable SSL Proxying → add a location with * for host and port to capture all traffic. 

Mobile device configuration (Android)

To configure Charles Proxy on Android, follow the same instructions for Wi-Fi, Manual setting, and port 8888. Go to Settings → Wi-Fi on your Android device.

• Hold the connected network to open the menu and select Modify Network.
• Choose Advanced Options → Proxy → Manual.
• Enter your computer's IP address as the Proxy Hostname and 8888 as the Port → Save.
• Open your mobile browser and go to chls.pro/ssl.
• Download the SSL Certificate and install it.

Note: Depending on your Android version, to install the SSL Certificate, you may need to go to Settings → Security → Credential Storage. For Android 7+ (Nougat and later), apps ignore user-installed certificates, so you need a custom Network Security Configuration in the app. Add a networkSecurityConfig file.

Setting up SSL proxying and certificate management

To avoid some of the most common friction points for new users, first we need to understand several crucial points. SSL proxying is essential because without it, there is no middle point that decrypts and re-encrypts traffic, which means that the encrypted requests will not become readable content.

SSL proxying enables traffic inspection for security analysis, debugging, and web scraping while masking user IP addresses. To use it, you must install and trust Charles Proxy’s Root Certificate on each device so it can securely intercept HTTPS traffic.

Charles generates unique certificates for each installation, and certificates should never be shared across devices. Once configured, all internet traffic passes through Charles before reaching the target server and returning to the device.

On macOS, install the certificate through Help → SSL Proxying → Install Charles Root Certificate, then set it to Always Trust in Keychain Access. On Windows, trust it through the system certificate settings.

To enable SSL proxying, go to Proxy → SSL Proxying Settings and enable SSL Proxying. Add domains by entering a host and port, or use * as a wildcard to capture all traffic.

Wildcards are useful for quick troubleshooting, but specifying domains is better for security, stability, performance, and reducing unnecessary traffic.

Inspecting and modifying HTTP/HTTPS traffic

After the setup, inspect and modify traffic, manipulate and debug data.

Inspecting traffic

Start by viewing requests and the real-time traffic, and inspecting elements. The Filter box is useful to filter traffic by domain, path, or content type. Click on a specific request to open the Request/Response Viewer in the main pane to view what the client sends and the server returns.

In the overview tab, you’ll find all details for each request in the session list. This includes request method, URL, HTTP headers, query parameters, cookies, and response body. The content viewers support JSON, XML, HTML, form data, and raw formats for Request and Response.

Using breakpoints for live editing

Right-click a request or domain and select Enable Breakpoints to intercept and modify a request before it reaches the server. This tells Charles to pause the request when it’s made, so you can manually edit the URL, body, or headers before continuing.

Rewrite rules for persistent modifications

Unlike breakpoints, which require manual edits, the Charles Proxy Rewrite Tool automatically modifies requests and responses based on defined rules. It’s useful for tasks like changing headers, injecting cookies, or editing API responses.

To set it up, go to Tools → Rewrite, enable Rewrite, create a new set, and add a Location with the target host and path. You can use wildcards (*) to match all endpoints on a host.

Map Remote and Map Local

Charles Proxy includes two traffic-mapping tools. Map Local serves a local file in place of a network response, making it useful for mocking APIs or testing frontend changes without a server. Map Remote redirects requests to another URL while keeping the original URL visible to the client, which is useful for routing staging traffic to a local server.

Both tools support wildcards (*) in the Host and Path fields for mapping entire domains or directories.

Bandwidth throttling and network simulation

Charles Proxy's throttling feature helps test application behavior, performance bottlenecks, and user experience under different network conditions. It restricts bandwidth and adds artificial latency to simulate slow or unreliable network conditions (3G, 4G, high latency). To do this, configure custom settings or use presets within the Proxy → Throttle Settings menu. You can enable throttling globally or per domain.

Developers and QA engineers can use throttling to test app behavior on slow or unstable networks, including loading states, timeouts, and mobile performance on cellular connections.

Keep in mind that throttling affects all traffic routed through Charles, so disable it after testing to avoid slowing normal browsing.

Practical use cases for Charles Proxy

Charles has a number of applications ranging from testing to security:

• API development and testing. Charles Proxy is widely used for inspecting API requests and responses in real time. Developers can verify headers, tokens, payloads, and error responses, as well as modify and replay requests for testing. Responses are automatically formatted for easier debugging.
• Web application debugging. Charles helps trace AJAX and REST API calls, inspect headers and response codes, monitor redirects, and troubleshoot caching or authentication issues. Tools like Structure view and Map Remote simplify backend testing in staging environments.
• Mobile app development. Charles provides visibility into iOS and Android network traffic that browser DevTools cannot access, helping developers debug APIs, caching, and mobile data handling.
• Security testing. Charles can inspect HTTPS traffic to verify secure handling of credentials, tokens, and other sensitive data. It is also useful for testing SSL implementations and server-side validation.
• Performance optimization. Charles identifies slow API calls, measures payload sizes, verifies compression, and simulates poor network conditions such as 2G/3G or high latency to test app stability, loading states, and error handling.

Using Charles Proxy for web scraping

Charles Proxy is a useful tool for web scraping because it helps discover API endpoints, understand request patterns, replicate browser requests in code, and troubleshoot complex scraping workflows. It makes web scraping more precise by exposing client-server communication and revealing the underlying data sources behind web applications.

• Endpoint discovery. Modern websites often load data through background API calls instead of static HTML. Charles Proxy can intercept this traffic to reveal hidden endpoints, headers, and JSON/XML payloads, making direct API access faster and more efficient than scraping rendered pages.
• Replicating browser requests. Charles captures headers, cookies, authentication tokens, and request parameters, allowing users to reproduce browser or mobile app requests. Captured traffic can also be exported as cURL commands for use in Python or other scraping tools.
• Identifying key headers. Important headers include authentication tokens, session cookies, User-Agent, Referer, and Content-Type. These are often required to maintain sessions and successfully mimic browser behavior.
• Determining required headers. Using the Compose tool, users can remove headers one by one and resend requests to identify which values are mandatory, including dynamic tokens or cookies needed for authentication.
• Exporting for scraping scripts. Requests can be exported from Charles as cURL commands or converted into Python, Node.js, or Postman code, making it easier to build accurate scraping scripts.
• Analyzing anti-bot protections. Charles Proxy helps analyze anti-bot systems by revealing security measures such as dynamic tokens, session cookies, and JavaScript-generated parameters hidden in HTTPS traffic. Understanding these mechanisms allows developers to replicate required requests more accurately and test anti-bot handling before implementing it in scraping scripts.
• Debugging scrapers with proxies. Charles Proxy can be used alongside rotating residential proxies to inspect scraping traffic in real time. It helps verify IP rotation, request headers, payloads, and response codes, making it easier to determine whether issues come from the scraper, proxy, or target site.

Configuring Charles with residential proxies

When set as an upstream proxy, Charles sits between the scraping script and the proxy provider. By entering the provider’s IP, port, and authentication details, users can monitor requests before they reach the target site, check sticky sessions, and detect issues like incorrect User-Agent values or leaked headers.

Here is a practical example of using Charles Proxy to: 1) Catch the request behind infinite scroll; 2) Identify pagination (after cursor); and 3) Copy as cURL and turn into Python.

1. Open the site

Go to the Reddit Python subreddit.

https://www.reddit.com/r/python/

https://www.reddit.com/r/python/

Open Charles Proxy and clear logs: 

• Top menu → Proxy → Clear Charles History. 
• Or press the broom icon in the toolbar.

In Charles, make sure that the Structure view is enabled (left sidebar tree view) and Recording is ON (red circle button at the top).

You should now see requests appearing live as the page loads.

2. Trigger infinite scroll

Scroll down slowly on the Reddit page.

In Charles:

• Look in the left sidebar (Structure tab).
• Expand https://www.reddit.com, then expand /r/python/.

You’re looking for requests like:

/r/python/.json?limit=25&after=t3_abc123

/r/python/.json?limit=25&after=t3_abc123

Note: Infinite-scroll requests usually appear right when new posts load. You can use the Charles filter: Bottom filter bar → type .json or type after=.

If you see too many requests, you can use the Sequence view to sort them chronologically and click the newest Reddit requests after scrolling.

3. Identify the API request

Click one of the .json requests in Charles. The request details panel appears on the right.

To see the Request URL, go to: 

• Request → Query String.
• Or the Overview tab. 

You’ll see something like:

https://www.reddit.com/r/python/.json?limit=25&after=t3_abc123

https://www.reddit.com/r/python/.json?limit=25&after=t3_abc123

To see the JSON response, go to: 

• Response → JSON Text.
• Or Response → Raw. 

You’ll see:

{
  "data": {
    "children": [...],
    "after": "t3_xyz789"
  }
}

{
  "data": {
    "children": [...],
    "after": "t3_xyz789"
  }
}

Note: 

• children: the posts returned.
• after: pagination cursor.
• The next request uses that after value.

Let’s check out an example flow:

Request 1:
limit=25


Response:
after=t3_xyz789


Request 2:
limit=25&after=t3_xyz789

Request 1:
limit=25


Response:
after=t3_xyz789


Request 2:
limit=25&after=t3_xyz789

That’s how infinite scrolling pagination works.

4. Check headers

In Charles, with the request selected, go to Request → Headers.

Expand Header Fields.

You usually only need:

user-agent: Mozilla/5.0
accept: application/json

user-agent: Mozilla/5.0
accept: application/json

Reddit’s public JSON endpoint is forgiving, so no auth is needed here. 

5. Copy as cURL

Right-click the request in Charles: Copy → cURL Request.

Charles copies the exact browser request.

For example:

curl 'https://www.reddit.com/r/python/.json?limit=25&after=t3_abc123' \
  -H 'user-agent: Mozilla/5.0' \
  -H 'accept: application/json'

curl 'https://www.reddit.com/r/python/.json?limit=25&after=t3_abc123' \
  -H 'user-agent: Mozilla/5.0' \
  -H 'accept: application/json'

This is useful because it reproduces the browser request exactly, and because you can test it in the terminal first, then convert it into Python.

6. Convert to Python

import requests

url = "https://www.reddit.com/r/python/.json"

headers = {
    "user-agent": "Mozilla/5.0",
    "accept": "application/json"
}

params = {
    "limit": 25
}

after = None

while True:
    if after:
        params["after"] = after

    res = requests.get(url, headers=headers, params=params)
    data = res.json()

    posts = data["data"]["children"]

    if not posts:
        break

    for post in posts:
        print(post["data"]["title"])

    after = data["data"]["after"]

    if not after:
        break

import requests

url = "https://www.reddit.com/r/python/.json"

headers = {
    "user-agent": "Mozilla/5.0",
    "accept": "application/json"
}

params = {
    "limit": 25
}

after = None

while True:
    if after:
        params["after"] = after

    res = requests.get(url, headers=headers, params=params)
    data = res.json()

    posts = data["data"]["children"]

    if not posts:
        break

    for post in posts:
        print(post["data"]["title"])

    after = data["data"]["after"]

    if not after:
        break

Note: Modern Reddit (and many other sites) may use different endpoints internally and/or add anti-bot protections. But this pattern is how the majority of scraping workflows start.

For production scraping, Decodo residential proxies are the top recommendation. They offer automatic IP rotation and integrate easily with Java HTTP clients. Decodo provides 115M+ ethically-sourced residential IPs across 195+ locations with the market's best response times, ensuring complete anonymity and bypassing geo-restrictions without hassle.

To set up Decodo residential proxies with rotating IPs:

1. Register or log in to the Decodo dashboard.
2. Navigate to Residential proxies, choose a subscription or start a 3-day free trial.
3. Go to Proxy setup.
4. Select a location or choose Random.
5. Set the rotating session type and choose a protocol (HTTP(S) or SOCKS5).
6. Choose the authentication type.
7. Download the generated endpoint and credentials or copy them into your scraper, browser, or software.

Troubleshooting common Charles Proxy issues

Because there are many spinning parts in how Charles works, you might run into an unexpected issue or two. Here are some of the most common ones and how to solve them.

SSL certificate errors

Charles uses a self-signed certificate to decrypt HTTPS traffic, which is blocked by default. Install and trust the Charles Root Certificate via Help → SSL Proxying → Install, then mark it as trusted in Keychain Access (macOS) or Trusted Root store (Windows).

Encrypted traffic is not readable

If responses appear as unreadable text, SSL Proxying likely isn’t enabled for that domain. Enable it in Proxy → SSL Proxying Settings and add the target host (or use * as a wildcard).

Expired certificates

Charles certificates expire after a year, causing connection issues. Reset and reinstall the certificate via Help → SSL Proxying → Reset Charles Root Certificate, then reinstall it.

SSL pinning blocks

Some secure apps reject Charles even with a trusted certificate due to SSL pinning. For your own apps, disable pinning during development. For third-party apps, bypassing it typically requires advanced techniques like device modification or instrumentation tools such as Frida.

Connection problems

If your internet stops working or no traffic appears in Charles, make sure the proxy is set to 127.0.0.1:8888. If Charles crashes, disable the proxy in your OS settings or run netsh winhttp reset proxy (Windows). Also, turn off VPNs or other proxies that may conflict.

Mobile device setup

For phones/tablets, use your computer’s local IP and make sure both devices are on the same Wi-Fi network. In Charles, go to Proxy → Access Control Settings and allow the phone’s IP.

HTTPS / SSL Issues

If HTTPS traffic stays encrypted, open chls.pro/ssl on the phone to install the Charles certificate. On iPhone: enable Full Trust for the certificate in Settings. Remember to add the domain to Charles SSL Proxying Settings.

Firewall issues

Make sure your firewall or antivirus allows Charles to accept connections on port 8888.

High CPU or memory usage

Charles can slow down if it records too much traffic. To avoid this: record only active sessions, exclude unnecessary hosts in Recording Settings, clear sessions regularly, and stop recording when not inspecting traffic.

SSL proxying slowdowns

Decrypting all HTTPS traffic uses a lot of CPU power, so enable SSL Proxying only for needed domains, and disable decryption for background/system traffic to improve speed.

Slow internet (“ghost” throttle)

Bandwidth throttling may still be enabled from testing, so check and disable Bandwidth Throttling when finished.

High-traffic limitations

Charles is best for inspecting specific API requests, not stress testing.

Integration with other tools

There are a handful of integrations that might help you build an optimized setup that fits your specifications:

Postman: building API collections

Charles traffic can be exported and reused in Postman for testing and automation. In Charles, right-click a request/session and export it as .chlsj (JSON). Then, convert the file into a Postman Collection. You can then run, edit, and automate tests using real captured API data.

Browser DevTools

DevTools shows browser-side details like the DOM, JavaScript, and tab requests, and Charles captures system-wide traffic, including other apps and background processes. Using both helps identify whether issues come from the frontend or backend.

cURL requests

Charles can instantly turn captured requests into terminal commands. Right-click a request and select Copy cURL Request to copy the full request. You can paste it into a terminal or scripts for quick testing and automation.

Load testing 

Charles can record real user sessions for stress testing. You can export sessions as HAR or XML files, and import them into tools like Apache JMeter or k6 to generate realistic load tests.

CI/CD pipelines

Recorded sessions can be reused in automated testing. This means that you can replay saved HTTP/HTTPS sessions in CI/CD workflows, mock external APIs without relying on live servers, and simulate errors, slow networks, or timeouts to test app reliability before deployment.

Proxy chaining

Proxy chaining lets Charles inspect traffic while routing requests through another proxy provider. Charles decrypts and analyzes traffic locally, then forwards requests through residential or rotating proxies. This helps debug geo-restricted or IP-blocked APIs without exposing your real IP. It’s also useful for checking whether proxy rotation, sticky sessions, or authentication are working correctly before deployment.

Charles Proxy alternatives

While Charles Proxy is widely used, some projects may benefit from alternatives with different strengths, such as lower-level packet analysis or platform-specific tools. Here are the top alternatives and how they compare.

Fiddler

This is a strong alternative to Charles for HTTP/HTTPS debugging. It’s best for .NET developers and advanced Windows debugging. Fiddler Classic is free and Windows-only, while Fiddler Everywhere is a paid, cross-platform version.

mitmproxy

This is a lightweight, terminal-based open-source proxy tool, best for automation, CI/CD, and command-line workflows. It comes with strong Python scripting support. It also supports HTTP/2 and WebSockets.

Proxyman

Proxyman is a macOS-native alternative, best suited for Apple users and mobile developers. It has a fast Swift-based interface, as well as easy SSL certificate management and iPhone traffic capture.

Wireshark

Wireshark is not exactly a direct Charles replacement, but rather a low-level packet analyzer. It captures raw network traffic and protocols beyond HTTP. It’s more complex than Charles for API debugging, but it’s also better for diagnosing packet loss and connectivity issues.

Browser DevTools

Built-in tools in browsers like Chrome and Firefox are great for frontend debugging and inspecting web requests. No setup is required, but these tools can’t capture traffic from mobile apps or system processes, which is something to keep in mind.

When projects grow into large-scale scraping or data collection, these tools are often paired with residential proxy networks. When the goal is data collection at scale rather than debugging, dedicated scraping infrastructure with residential proxies and anti-bot handling are more appropriate than a debugging proxy, and Decodo residential proxies are purpose-built for production scraping.

Final thoughts

Charles Proxy remains one of the most versatile tools for inspecting and debugging network traffic. It helps developers, QA testers, and API researchers analyze requests, troubleshoot issues, and understand application behavior in detail. Take advantage of the free trial to experiment with Charles Proxy and understand your application's network behavior in detail.

Charles is mainly designed for development and debugging, not large-scale production scraping or high-volume automation. Heavy workloads can create performance and hardware limitations. Therefore, use Charles to analyze and map requests, and use dedicated scraping APIs and residential proxies for large-scale execution, IP rotation, and geo-unblocking. Charles is especially valuable during the discovery and testing phase, where visibility into requests and responses is critical before scaling a project further.