C++ Web Scraping: A Practical Guide for Performance-Critical Projects

TL;DR

• Use C++ when request throughput, memory control, or native C++ integration is the real bottleneck.
• Start with CPR and libxml2 for most projects, then switch parsers based on the target format.
• Treat JavaScript rendering, proxy rotation, and anti-bot friction as delivery problems, not parsing problems.
• Offload browser rendering and unblock work to a managed API when maintenance starts costing more than the data.

Is C++ a good choice for web scraping?

If you're comparing languages for scraping, it isn't whether C++ can do it. It can. The better question is what you need from the scraper: lower CPU cost per request, tighter memory control, predictable latency, or faster iteration. That's where C++ either makes a strong case or quickly becomes more work than it's worth. If you want a broader language trade-off, it also helps to compare it with other options in a guide to the best programming languages for web scraping. And if you're still separating collection from extraction, a quick refresher on web crawling vs. web scraping makes the language choice easier to place.

Where C++ wins

• Throughput per core. A tight libcurl-plus-libxml2 loop can keep requests and parsing in the same address space, which eliminates interpreter overhead, IPC (inter-process communication between separate processes), and garbage-collection pauses. That matters when you're pushing very large volumes through 1 machine or trying to squeeze more work out of each core.
• Memory footprint: C++ gives you much more control over allocation patterns, parser lifetimes, and connection reuse. That's useful when the scraper runs inside a memory-capped container, on edge hardware, or anywhere per-process memory cost matters.
• Deterministic latency. There's no garbage collector (automatic memory cleanup) and no JIT (just-in-time) compiler warm-up. That makes C++ attractive for latency-sensitive pipelines where predictable behavior matters more than developer ergonomics. 
• Native interop. If the rest of the system is already in C++, scraping in the same language removes 1 more boundary. That matters in trading systems, game backends, embedded analytics, and other pipelines where switching to Python or Node adds extra operational friction.

Where C++ loses

• No first-class scraping framework. No C++ equivalent to tools such as Scrapy gives you middleware, retries, item pipelines, selector helpers, and concurrency patterns out of the box. In C++, you assemble the stack yourself.
• No strong native headless browser story. You can drive Chrome from C++, but the ecosystem is much thinner than Playwright or Puppeteer. That becomes a real problem once the target depends on client-side rendering, browser fingerprints, or modern anti-bot checks.
• Build complexity. Every dependency adds more setup: CMake config, package manager choices, ABI compatibility (whether compiled libraries can work together correctly), and cross-platform differences. That's manageable in a mature C++ codebase, but it's slower than spinning up a scraper in Python or Node.
• Smaller scraping community. There are fewer tutorials, fewer examples, and fewer ready-made integrations than you get in Python or JavaScript. That doesn't make C++ a bad choice. It just means you'll solve more of the plumbing yourself.

Recommendation

Use C++ when scraping is part of an existing C++ system, or when per-request CPU and memory costs are high enough to justify the extra setup. That’s where the language pays you back.

For one-off jobs, exploratory scraping, MVPs, and anything that changes often, Python or Node will usually ship faster and cost less to maintain. That's the real tradeoff: C++ gives you tighter runtime control, but you earn it with more engineering overhead.

Setting up your C++ web scraping environment

Before you write any scraping code, set up the project's tools in C++, which means more than just installing a library. You need a compiler to build the code, a build system to organize the project, and a package manager to install dependencies.

For this guide, let’s keep the setup simple:

• Use a modern C++ compiler
• Use CMake to build the project
• Use 1 package manager consistently
• Install the scraping libraries only after those pieces are working

That order matters. Without CMake or a configured compiler, the libraries won't help because the project won't build.

What each tool does

Before the setup steps, it helps to know what each tool is for:

• The compiler turns your C++ code into an executable program
• CMake tells the compiler how to build the project and link libraries
• The package manager installs libraries such as cpr, libxml2, and pugixml

If you’re new to C++, think of CMake as the piece that ties the whole project together. Once the project grows beyond 1 source file, building it by hand gets messy fast.

Toolchain prerequisites

• Use a modern compiler. A practical baseline is g++ 11+, clang 14+, or MSVC 19.30+. That gives you C++17 features such as std::filesystem and structured bindings.
• Use CMake 3.20+. Once your project has more than 1 source file, building it by hand becomes tedious. CMake keeps the build organized and lets you use find_package() instead of manually writing include paths and linker flags.
• Pick 1 package manager and stay consistent. Your main options are vcpkg, Conan, or your OS's package manager (e.g., apt, dnf, or Homebrew). Mixing them in 1 project usually creates more setup problems than it solves.
• Choose the path that best matches your environment. vcpkg works well across platforms, Conan gives you more control over dependency versions, and system package managers are fine for simpler local setups.

Windows setup with vcpkg

On Windows, the cleanest beginner setup is:

1. Install Visual Studio with C++

Start by installing Visual Studio 2022. During setup, make sure you select the Desktop development with C++ workload.

That step gives you the core things a C++ project needs on Windows:

• the MSVC compiler
• the Windows SDK
• the C++ build tools

Without that workload, Visual Studio may open the project, but it will not have the compiler and build tools needed actually to compile C++ code. 

2. Install Git

Install Git next. You need Git because the easiest way to get vcpkg is to clone it from GitHub.

After installing Git, open a terminal and run git --version. If you see a version number, Git is working.

3. Install CMake and vcpkg

Now install CMake.

CMake is the tool that reads your CMakeLists.txt file and builds your project. In simple terms, it is what tells your computer how to compile your scraper. During installation, make sure you choose the option to add CMake to PATH.

After installing it, open a new terminal and run cmake --version.

If that command doesn't work, it usually means it isn’t installed yet or wasn't added to PATH during installation; stop there and fix CMake before going further. The scraper will not build without it.

Once CMake is installed, clone and bootstrap vcpkg: 

git clone https://github.com/microsoft/vcpkg.git
cd vcpkg
.\bootstrap-vcpkg.bat
.\vcpkg install curl libxml2 cpr pugixml
.\vcpkg integrate install

git clone https://github.com/microsoft/vcpkg.git
cd vcpkg
.\bootstrap-vcpkg.bat
.\vcpkg install curl libxml2 cpr pugixml
.\vcpkg integrate install

That last command wires vcpkg into MSBuild projects, which makes Visual Studio integration much smoother. The vcpkg package registry currently includes ports for curl, libxml2, cpr, and pugixml.

If you want the project to stay reproducible across teammates and CI, add a vcpkg.json file in the repo root. Microsoft recommends manifest mode for most users, and vcpkg.json is the file that drives it. That's the easiest way to avoid the usual “works on my machine” dependency mismatch.

macOS setup with Homebrew

If you need to build the same project on macOS, Homebrew covers the core packages:

brew install cmake curl libxml2 pugixml cpr

brew install cmake curl libxml2 pugixml cpr

That works, but there’s 1 extra detail to watch out for. Homebrew marks libxml2 as keg-only, which means macOS can still prefer Apple’s bundled copy unless you point pkg-config or CMake at the Homebrew install explicitly. On Apple Silicon, it’s also worth checking that you’re building for arm64 so you don't accidentally compile an x86_64 binary via Rosetta.

Linux setup with apt or dnf

On Ubuntu or Debian, the base setup is straightforward:

sudo apt install cmake libcurl4-openssl-dev libxml2-dev libpugixml-dev

sudo apt install cmake libcurl4-openssl-dev libxml2-dev libpugixml-dev

Those packages cover libcurl, libxml2, pugixml, and CMake. CPR is the 1 library that may require extra work depending on the distro and release. If it’s not available in your repos, install it with vcpkg or build it from source with CMake. CPR’s own project documents the source-build path and its integration with CMake.

On RHEL-based systems, the equivalent package names usually look like this:

sudo dnf install cmake libcurl-devel libxml2-devel pugixml-devel

sudo dnf install cmake libcurl-devel libxml2-devel pugixml-devel

The package names vary a bit across distributions, but the overall pattern remains the same: install the HTTP library, the parser, and CMake first, then let your build system link them cleanly.

What your minimal CMakeLists.txt should contain

CMakeLists.txt is the file that tells CMake how to build your project. It describes the project name, the C++ version to use, which source files to compile, and which libraries to link. Without it, CMake has no idea how your scraper should be built. 

You don’t need a long CMakeLists.txt file to get started. At minimum, it should:

• Set cmake_minimum_required
• Declare the project name
• Set CMAKE_CXX_STANDARD to 17 or 20
• Call find_package() for CURL, LibXml2, cpr, and pugixml
• Link those libraries with modern imported targets instead of raw -l flags

The targets you want are:

• CURL::libcurl
• LibXml2::LibXml2
• cpr::cpr
• pugixml::pugixml

CMake’s official modules document CURL::libcurl and LibXml2::LibXml2, and CPR documents cpr::cpr in its CMake usage. This setup keeps the build cleaner because CMake handles the include paths and linker settings for you once the libraries are installed correctly. In other words, instead of manually wiring everything together, you let the build system do the work. 

If cURL semantics are still new, this is also a good point to brush up on sending HTTP headers with cURL, since the same principles apply directly to libcurl.

C++ web scraping libraries and tools

Once your build setup is ready, the next choice is the stack itself. In C++, that usually means picking 1 HTTP client library and 1 parser. The right combination depends less on popularity and more on what kind of scraping you’re doing: static HTML pages, XML feeds, malformed markup, or real-time endpoints. If you want more parser background before choosing, it helps to explore our guide to choosing the best parser. And if you’re weighing query styles, XPath vs. CSS selectors become relevant very quickly once you compare libxml2 against tools that lean more toward CSS-style querying.

HTTP client libraries

• libcurl. This is the most established option. It’s the underlying HTTP engine for many C++ networking stacks, and it supports proxies, cookies, authentication, HTTP/2, WebSocket, and a long list of protocols. It can also support HTTP/3 when built with the required QUIC and TLS dependencies. The tradeoff is ergonomics. The API is lower-level, more verbose, and callback-heavy, but it's still the safest production choice when you need maximum control.
• CPR. This is the easiest default for most new C++ scrapers. CPR wraps libcurl with a cleaner C++ interface inspired by Python’s requests, while still relying on libcurl underneath. CPR 1.10.0 and later require a C++17-compatible compiler, while older CPR versions may still support C++11 and usually give you the same basic request flow with much less code than raw libcurl. 
• Boost.Beast. This is the right pick when you need asynchronous networking, WebSockets, or more advanced streaming patterns. Beast sits on top of Boost.Asio and is much better suited to real-time scraping workloads like live feeds than a simple request-response wrapper. The tradeoff is complexity. It’s powerful, but it has a steeper learning curve than CPR or libcurl.
• cpp-httplib. This is a lightweight option for small tools. It's header-only and easy to drop into a project, which makes it attractive for CI jobs, one-file utilities, or environments where you want very few dependencies. But it’s not a full replacement for libcurl. The project itself notes that it doesn’t support all the advanced options available in libcurl, and it’s not the first choice when proxies, large-scale connection handling, or advanced protocol support matter.

HTML and XML parsers

• libxml2. This is still the standard when you need XPath (a query language for selecting nodes and data from HTML or XML documents). It's fast, mature, and it supports XPath 1.0 directly. The downside is that the API is pointer-heavy and less beginner-friendly, and its HTML handling isn’t as forgiving as browser-style parsers when markup is badly broken.
• pugixml. This is a cleaner, easier-to-use parser for well-formed XML. It's a strong fit for sitemaps, RSS feeds, and structured API responses. It’s less ideal for messy real-world HTML, but much nicer to work with than libxml2 if the target format is actually XML.
• Gumbo. This can still be useful for broken HTML, but it needs a caveat. Gumbo is an HTML5 parser written in C, designed to recover the way browsers do. Still, Google archived the original repository in January 2026, so treat it as a legacy option rather than an actively maintained default. 
• Lexbor: This is a newer parser worth evaluating when parser performance matters. Lexbor is a pure-C HTML engine that includes HTML5 parsing and CSS selector support, which gives it an edge over Gumbo if you want browser-style parsing plus built-in querying. It's not as entrenched as libxml2, but it's worth a look when parser speed or recovery from malformed HTML becomes the bottleneck.

Recommended pairings

If you don't want to overthink the first version of your scraper, these combinations are the most practical starting points:

• CPR + libxml2. Best default for most C++ web scrapers. You get a cleaner HTTP API and proven XPath parsing.
• CPR + pugixml. Best when the target is XML-first, such as sitemaps, RSS, or structured API output.
• libcurl + Gumbo or libcurl + Lexbor. Better when the target HTML is messy, and you want a more actively maintained parser. Gumbo can still work for legacy projects, but the original Google repository is now archived. 
• Boost.Beast + libxml2. Better when you need async I/O, WebSockets, or real-time feeds and still want XPath for extraction.

The simplest way to choose is this: start with CPR + libxml2 unless you already know the target is XML-only, badly broken, or streaming in real time. That pairing gives most beginners the best balance between readable code and reliable parsing.

Building a basic C++ web scraper

The easiest way to understand C++ web scraping is to build 1 small scraper from start to finish. For this guide, use books.toscrape.com. It's public, stable, and simple enough for a first scraper. The goal is to end with a working binary that fetches a page, parses the HTML, extracts book data, and follows the next page link.

If DOM extraction is new to you, it also helps to review what parsing is before wiring XPath into the scraper.

Step 1: Create the project files

Start with a minimal structure:

• CMakeLists.txt
• main.cpp

At this stage, 1 source file is enough. The point is to prove the full request → parse → extract flow before splitting code into separate modules.

Your CMakeLists.txt should look like this:

cmake_minimum_required(VERSION 3.20)
project(BookScraper LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_package(cpr CONFIG REQUIRED)
find_package(LibXml2 REQUIRED)

add_executable(BookScraper main.cpp)

target_link_libraries(BookScraper PRIVATE
    cpr::cpr
    LibXml2::LibXml2
)

cmake_minimum_required(VERSION 3.20)
project(BookScraper LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_package(cpr CONFIG REQUIRED)
find_package(LibXml2 REQUIRED)

add_executable(BookScraper main.cpp)

target_link_libraries(BookScraper PRIVATE
    cpr::cpr
    LibXml2::LibXml2
)

This does the basics:

• Sets the project to C++ 17
• Finds CPR
• Finds libxml2
• Builds main.cpp into 1 executable called BookScraper

Step 2: Create main.cpp and define the data model

Now, create main.cpp and start with the headers and a small data model:

#include <iostream>
#include <string>
#include <vector>
#include <cpr/cpr.h>
#include <libxml/HTMLparser.h>
#include <libxml/xpath.h>
#include <libxml/xpathInternals.h>


struct Book {
    std::string title;
    std::string price;
    std::string url;
};

#include <iostream>
#include <string>
#include <vector>
#include <cpr/cpr.h>
#include <libxml/HTMLparser.h>
#include <libxml/xpath.h>
#include <libxml/xpathInternals.h>


struct Book {
    std::string title;
    std::string price;
    std::string url;
};

For books.toscrape.com, those 3 fields are enough for a first working scraper:

• title
• price
• product URL

Step 3: Issue the HTTP request

Use CPR to fetch the page. It's much easier to read than raw libcurl for a first build.

int main() {
    auto response = cpr::Get(
        cpr::Url{"https://books.toscrape.com/catalogue/page-1.html"},
        cpr::Header{{"User-Agent", "Mozilla/5.0"}}
    );

    if (response.status_code != 200) {
        std::cerr << "Request failed with status " << response.status_code << '\n';
        return 1;
    }

    std::cout << "Downloaded " << response.text.size() << " bytes\n";
}

int main() {
    auto response = cpr::Get(
        cpr::Url{"https://books.toscrape.com/catalogue/page-1.html"},
        cpr::Header{{"User-Agent", "Mozilla/5.0"}}
    );

    if (response.status_code != 200) {
        std::cerr << "Request failed with status " << response.status_code << '\n';
        return 1;
    }

    std::cout << "Downloaded " << response.text.size() << " bytes\n";
}

That does 3 important things right away:

• Sends a real HTTP request
• Checks the status code
• Stops early if the request fails

Step 4: Parse the HTML safely

Once you have the response body, hand it to libxml2’s HTML parser. This is the step where the raw HTML becomes a document you can query.

htmlDocPtr doc = htmlReadMemory(
    response.text.c_str(),
    static_cast<int>(response.text.size()),
    "https://books.toscrape.com/catalogue/page-1.html",
    nullptr,
    HTML_PARSE_NOERROR | HTML_PARSE_NOWARNING
);

if (!doc) {
    std::cerr << "Failed to parse HTML\n";
    return 1;
}

xmlXPathContextPtr xpathCtx = xmlXPathNewContext(doc);
if (!xpathCtx) {
    std::cerr << "Failed to create XPath context\n";
    xmlFreeDoc(doc);
    return 1;
}

htmlDocPtr doc = htmlReadMemory(
    response.text.c_str(),
    static_cast<int>(response.text.size()),
    "https://books.toscrape.com/catalogue/page-1.html",
    nullptr,
    HTML_PARSE_NOERROR | HTML_PARSE_NOWARNING
);

if (!doc) {
    std::cerr << "Failed to parse HTML\n";
    return 1;
}

xmlXPathContextPtr xpathCtx = xmlXPathNewContext(doc);
if (!xpathCtx) {
    std::cerr << "Failed to create XPath context\n";
    xmlFreeDoc(doc);
    return 1;
}

At this point, you have what you need for XPath:

• a parsed HTML document
• an XPath context

Step 5: Extract the book data with XPath

Now query the page for book cards and extract the fields from each one.

On books.toscrape.com, the useful structure is:

• article.product_pod for each book card
• .//h3/a for the title and relative link
• .//p[contains(@class, 'price_color')] for the price

A working extraction loop looks like this:

std::vector<Book> books;

xmlXPathObjectPtr cards = xmlXPathEvalExpression(
    BAD_CAST "//article[contains(@class, 'product_pod')]",
    xpathCtx
);

if (cards && cards->nodesetval) {
    for (int i = 0; i < cards->nodesetval->nodeNr; ++i) {
        xmlNodePtr card = cards->nodesetval->nodeTab[i];
        xmlXPathContextPtr cardCtx = xmlXPathNewContext(doc);
        cardCtx->node = card;

        Book book;

        xmlXPathObjectPtr titleObj = xmlXPathEvalExpression(BAD_CAST ".//h3/a", cardCtx);
        if (titleObj && titleObj->nodesetval && titleObj->nodesetval->nodeNr > 0) {
            xmlNodePtr linkNode = titleObj->nodesetval->nodeTab[0];

            xmlChar* titleAttr = xmlGetProp(linkNode, BAD_CAST "title");
            xmlChar* hrefAttr = xmlGetProp(linkNode, BAD_CAST "href");

            if (titleAttr) {
                book.title = reinterpret_cast<const char*>(titleAttr);
                xmlFree(titleAttr);
            }

            if (hrefAttr) {
                book.url = "https://books.toscrape.com/catalogue/" +
                           std::string(reinterpret_cast<const char*>(hrefAttr));
                xmlFree(hrefAttr);
            }
        }

        xmlXPathObjectPtr priceObj = xmlXPathEvalExpression(
            BAD_CAST ".//p[contains(@class, 'price_color')]",
            cardCtx
        );

        if (priceObj && priceObj->nodesetval && priceObj->nodesetval->nodeNr > 0) {
            xmlChar* priceText = xmlNodeGetContent(priceObj->nodesetval->nodeTab[0]);
            if (priceText) {
                book.price = reinterpret_cast<const char*>(priceText);
                xmlFree(priceText);
            }
        }

        books.push_back(book);

        if (titleObj) xmlXPathFreeObject(titleObj);
        if (priceObj) xmlXPathFreeObject(priceObj);
        xmlXPathFreeContext(cardCtx);
    }
}

if (cards) xmlXPathFreeObject(cards);

std::vector<Book> books;

xmlXPathObjectPtr cards = xmlXPathEvalExpression(
    BAD_CAST "//article[contains(@class, 'product_pod')]",
    xpathCtx
);

if (cards && cards->nodesetval) {
    for (int i = 0; i < cards->nodesetval->nodeNr; ++i) {
        xmlNodePtr card = cards->nodesetval->nodeTab[i];
        xmlXPathContextPtr cardCtx = xmlXPathNewContext(doc);
        cardCtx->node = card;

        Book book;

        xmlXPathObjectPtr titleObj = xmlXPathEvalExpression(BAD_CAST ".//h3/a", cardCtx);
        if (titleObj && titleObj->nodesetval && titleObj->nodesetval->nodeNr > 0) {
            xmlNodePtr linkNode = titleObj->nodesetval->nodeTab[0];

            xmlChar* titleAttr = xmlGetProp(linkNode, BAD_CAST "title");
            xmlChar* hrefAttr = xmlGetProp(linkNode, BAD_CAST "href");

            if (titleAttr) {
                book.title = reinterpret_cast<const char*>(titleAttr);
                xmlFree(titleAttr);
            }

            if (hrefAttr) {
                book.url = "https://books.toscrape.com/catalogue/" +
                           std::string(reinterpret_cast<const char*>(hrefAttr));
                xmlFree(hrefAttr);
            }
        }

        xmlXPathObjectPtr priceObj = xmlXPathEvalExpression(
            BAD_CAST ".//p[contains(@class, 'price_color')]",
            cardCtx
        );

        if (priceObj && priceObj->nodesetval && priceObj->nodesetval->nodeNr > 0) {
            xmlChar* priceText = xmlNodeGetContent(priceObj->nodesetval->nodeTab[0]);
            if (priceText) {
                book.price = reinterpret_cast<const char*>(priceText);
                xmlFree(priceText);
            }
        }

        books.push_back(book);

        if (titleObj) xmlXPathFreeObject(titleObj);
        if (priceObj) xmlXPathFreeObject(priceObj);
        xmlXPathFreeContext(cardCtx);
    }
}

if (cards) xmlXPathFreeObject(cards);

Up to this point, we've only been fetching the page content. Now we're extracting the structured information from it. 

Step 6: Print the results and clean up

Before adding storage or pagination, print the results and make sure the selectors are working:

for (const auto& book : books) {
    std::cout << "Title: " << book.title << '\n';
    std::cout << "Price: " << book.price << '\n';
    std::cout << "URL: " << book.url << '\n';
    std::cout << "-----\n";
}

xmlXPathFreeContext(xpathCtx);
xmlFreeDoc(doc);
xmlCleanupParser();

return 0;

for (const auto& book : books) {
    std::cout << "Title: " << book.title << '\n';
    std::cout << "Price: " << book.price << '\n';
    std::cout << "URL: " << book.url << '\n';
    std::cout << "-----\n";
}

xmlXPathFreeContext(xpathCtx);
xmlFreeDoc(doc);
xmlCleanupParser();

return 0;

Don't skip this step. It's much easier to catch broken XPath here than after adding pagination or file output.

Step 7: Build the binary

From the project root, configure and build with CMake. If you installed dependencies with vcpkg, include the vcpkg toolchain file. Replace C:/path/to/vcpkg with the real path to your vcpkg folder:

cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=C:/path/to/vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Release

cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=C:/path/to/vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build build --config Release

If everything is linked correctly, CMake will generate the build files and compile the scraper into an executable.

Step 8: Run it and validate the output

Run the executable from the output folder. On Windows with Visual Studio generators, that's often under build/Release or build/Debug.

A successful run should show output like this:

Downloaded 50469 bytes
Title: A Light in the Attic
Price: £51.77
URL: https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
-----

Downloaded 50469 bytes
Title: A Light in the Attic
Price: £51.77
URL: https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
-----

Before moving on, check that:

• Titles aren't empty
• Prices are present
• URLs are resolved correctly
• Multiple book cards are being extracted, not just the first one

Step 9: Add basic pagination

Once page 1 works, add the next link so the scraper can move through the catalog automatically. On books.toscrape.com, that link appears inside li.next a. The first version only needs a simple loop:

• Fetch the current page
• Parse the HTML
• Extract the books
• Find the next-page link
• Stop when there's no next link

It's also worth adding a maximum page limit so the scraper can't run forever if the selector breaks.

Here's a working version that keeps scraping until there's no next page or until it reaches a simple safety limit: 

#include <iostream>
#include <string>
#include <vector>
#include <cpr/cpr.h>
#include <libxml/HTMLparser.h>
#include <libxml/xpath.h>
#include <libxml/xpathInternals.h>

struct Book {
    std::string title;
    std::string price;
    std::string url;
};

int main() {
    std::vector<Book> allBooks;

    std::string currentUrl = "https://books.toscrape.com/catalogue/page-1.html";
    int maxPages = 3;
    int pageCount = 0;

    while (!currentUrl.empty() && pageCount < maxPages) {
        auto response = cpr::Get(
            cpr::Url{currentUrl},
            cpr::Header{{"User-Agent", "Mozilla/5.0"}}
        );

        if (response.status_code != 200) {
            std::cerr << "Request failed with status " << response.status_code << '\n';
            break;
        }

        std::cout << "Scraping: " << currentUrl << '\n';

        htmlDocPtr doc = htmlReadMemory(
            response.text.c_str(),
            static_cast<int>(response.text.size()),
            currentUrl.c_str(),
            nullptr,
            HTML_PARSE_NOERROR | HTML_PARSE_NOWARNING
        );

        if (!doc) {
            std::cerr << "Failed to parse HTML\n";
            break;
        }

        xmlXPathContextPtr xpathCtx = xmlXPathNewContext(doc);
        if (!xpathCtx) {
            std::cerr << "Failed to create XPath context\n";
            xmlFreeDoc(doc);
            break;
        }

        xmlXPathObjectPtr cards = xmlXPathEvalExpression(
            BAD_CAST "//article[contains(@class, 'product_pod')]",
            xpathCtx
        );

        if (cards && cards->nodesetval) {
            for (int i = 0; i < cards->nodesetval->nodeNr; ++i) {
                xmlNodePtr card = cards->nodesetval->nodeTab[i];
                xmlXPathContextPtr cardCtx = xmlXPathNewContext(doc);
                cardCtx->node = card;

                Book book;

                xmlXPathObjectPtr titleObj = xmlXPathEvalExpression(BAD_CAST ".//h3/a", cardCtx);
                if (titleObj && titleObj->nodesetval && titleObj->nodesetval->nodeNr > 0) {
                    xmlNodePtr linkNode = titleObj->nodesetval->nodeTab[0];

                    xmlChar* titleAttr = xmlGetProp(linkNode, BAD_CAST "title");
                    xmlChar* hrefAttr = xmlGetProp(linkNode, BAD_CAST "href");

                    if (titleAttr) {
                        book.title = reinterpret_cast<const char*>(titleAttr);
                        xmlFree(titleAttr);
                    }

                    if (hrefAttr) {
                        book.url = "https://books.toscrape.com/catalogue/" +
                                   std::string(reinterpret_cast<const char*>(hrefAttr));
                        xmlFree(hrefAttr);
                    }
                }

                xmlXPathObjectPtr priceObj = xmlXPathEvalExpression(
                    BAD_CAST ".//p[contains(@class, 'price_color')]",
                    cardCtx
                );

                if (priceObj && priceObj->nodesetval && priceObj->nodesetval->nodeNr > 0) {
                    xmlChar* priceText = xmlNodeGetContent(priceObj->nodesetval->nodeTab[0]);
                    if (priceText) {
                        book.price = reinterpret_cast<const char*>(priceText);
                        xmlFree(priceText);
                    }
                }

                allBooks.push_back(book);

                if (titleObj) xmlXPathFreeObject(titleObj);
                if (priceObj) xmlXPathFreeObject(priceObj);
                xmlXPathFreeContext(cardCtx);
            }
        }

        if (cards) xmlXPathFreeObject(cards);

        xmlXPathObjectPtr nextObj = xmlXPathEvalExpression(
            BAD_CAST "//li[contains(@class, 'next')]/a",
            xpathCtx
        );

        if (nextObj && nextObj->nodesetval && nextObj->nodesetval->nodeNr > 0) {
            xmlNodePtr nextNode = nextObj->nodesetval->nodeTab[0];
            xmlChar* hrefAttr = xmlGetProp(nextNode, BAD_CAST "href");

            if (hrefAttr) {
                currentUrl = "https://books.toscrape.com/catalogue/" +
                             std::string(reinterpret_cast<const char*>(hrefAttr));
                xmlFree(hrefAttr);
            } else {
                currentUrl.clear();
            }
        } else {
            currentUrl.clear();
        }

        if (nextObj) xmlXPathFreeObject(nextObj);

        xmlXPathFreeContext(xpathCtx);
        xmlFreeDoc(doc);

        pageCount++;
    }

    xmlCleanupParser();

    for (const auto& book : allBooks) {
        std::cout << "Title: " << book.title << '\n';
        std::cout << "Price: " << book.price << '\n';
        std::cout << "URL: " << book.url << '\n';
        std::cout << "-----\n";
    }

    return 0;
}

#include <iostream>
#include <string>
#include <vector>
#include <cpr/cpr.h>
#include <libxml/HTMLparser.h>
#include <libxml/xpath.h>
#include <libxml/xpathInternals.h>

struct Book {
    std::string title;
    std::string price;
    std::string url;
};

int main() {
    std::vector<Book> allBooks;

    std::string currentUrl = "https://books.toscrape.com/catalogue/page-1.html";
    int maxPages = 3;
    int pageCount = 0;

    while (!currentUrl.empty() && pageCount < maxPages) {
        auto response = cpr::Get(
            cpr::Url{currentUrl},
            cpr::Header{{"User-Agent", "Mozilla/5.0"}}
        );

        if (response.status_code != 200) {
            std::cerr << "Request failed with status " << response.status_code << '\n';
            break;
        }

        std::cout << "Scraping: " << currentUrl << '\n';

        htmlDocPtr doc = htmlReadMemory(
            response.text.c_str(),
            static_cast<int>(response.text.size()),
            currentUrl.c_str(),
            nullptr,
            HTML_PARSE_NOERROR | HTML_PARSE_NOWARNING
        );

        if (!doc) {
            std::cerr << "Failed to parse HTML\n";
            break;
        }

        xmlXPathContextPtr xpathCtx = xmlXPathNewContext(doc);
        if (!xpathCtx) {
            std::cerr << "Failed to create XPath context\n";
            xmlFreeDoc(doc);
            break;
        }

        xmlXPathObjectPtr cards = xmlXPathEvalExpression(
            BAD_CAST "//article[contains(@class, 'product_pod')]",
            xpathCtx
        );

        if (cards && cards->nodesetval) {
            for (int i = 0; i < cards->nodesetval->nodeNr; ++i) {
                xmlNodePtr card = cards->nodesetval->nodeTab[i];
                xmlXPathContextPtr cardCtx = xmlXPathNewContext(doc);
                cardCtx->node = card;

                Book book;

                xmlXPathObjectPtr titleObj = xmlXPathEvalExpression(BAD_CAST ".//h3/a", cardCtx);
                if (titleObj && titleObj->nodesetval && titleObj->nodesetval->nodeNr > 0) {
                    xmlNodePtr linkNode = titleObj->nodesetval->nodeTab[0];

                    xmlChar* titleAttr = xmlGetProp(linkNode, BAD_CAST "title");
                    xmlChar* hrefAttr = xmlGetProp(linkNode, BAD_CAST "href");

                    if (titleAttr) {
                        book.title = reinterpret_cast<const char*>(titleAttr);
                        xmlFree(titleAttr);
                    }

                    if (hrefAttr) {
                        book.url = "https://books.toscrape.com/catalogue/" +
                                   std::string(reinterpret_cast<const char*>(hrefAttr));
                        xmlFree(hrefAttr);
                    }
                }

                xmlXPathObjectPtr priceObj = xmlXPathEvalExpression(
                    BAD_CAST ".//p[contains(@class, 'price_color')]",
                    cardCtx
                );

                if (priceObj && priceObj->nodesetval && priceObj->nodesetval->nodeNr > 0) {
                    xmlChar* priceText = xmlNodeGetContent(priceObj->nodesetval->nodeTab[0]);
                    if (priceText) {
                        book.price = reinterpret_cast<const char*>(priceText);
                        xmlFree(priceText);
                    }
                }

                allBooks.push_back(book);

                if (titleObj) xmlXPathFreeObject(titleObj);
                if (priceObj) xmlXPathFreeObject(priceObj);
                xmlXPathFreeContext(cardCtx);
            }
        }

        if (cards) xmlXPathFreeObject(cards);

        xmlXPathObjectPtr nextObj = xmlXPathEvalExpression(
            BAD_CAST "//li[contains(@class, 'next')]/a",
            xpathCtx
        );

        if (nextObj && nextObj->nodesetval && nextObj->nodesetval->nodeNr > 0) {
            xmlNodePtr nextNode = nextObj->nodesetval->nodeTab[0];
            xmlChar* hrefAttr = xmlGetProp(nextNode, BAD_CAST "href");

            if (hrefAttr) {
                currentUrl = "https://books.toscrape.com/catalogue/" +
                             std::string(reinterpret_cast<const char*>(hrefAttr));
                xmlFree(hrefAttr);
            } else {
                currentUrl.clear();
            }
        } else {
            currentUrl.clear();
        }

        if (nextObj) xmlXPathFreeObject(nextObj);

        xmlXPathFreeContext(xpathCtx);
        xmlFreeDoc(doc);

        pageCount++;
    }

    xmlCleanupParser();

    for (const auto& book : allBooks) {
        std::cout << "Title: " << book.title << '\n';
        std::cout << "Price: " << book.price << '\n';
        std::cout << "URL: " << book.url << '\n';
        std::cout << "-----\n";
    }

    return 0;
}

This is the simplest pagination pattern: follow the next link until it disappears. That works well for page-based sites like books.toscrape.com. For more complex targets, such as offset, cursor, or infinite-scroll pagination, see the guide on handling web scraping pagination.

What the first working version proves

Your first C++ scraper doesn't yet need concurrency, storage, retries, or proxy rotation. It only needs to prove that the full pipeline works. Once that works, you have a real base to build on. Then it makes sense to add storage, stronger error handling, retries, and more production-ready request logic.

Exporting and storing scraped data

Once the scraper is pulling clean records, the next question is where those records should go. The right answer depends on the volume, downstream use, and the level of structure you need after the crawl finishes. In C++, it helps to keep this simple at first: start with flat files, move to JSON when the shape gets richer, and write directly to a database only when the data needs to be queried or joined immediately. For a broader storage overview, it also helps to read how to save scraped data to CSV, Excel, and databases.

Option 1: CSV with std::ofstream

Use CSV when the schema is small, stable, and tabular. That's usually the right first step for page-level extracts like title, price, URL, rating, and availability.

CSV is a good fit when:

• You want something easy to inspect
• The fields are mostly scalar values
• The output will go to Excel, pandas, or a BI tool later

A minimal example looks like this:

#include <fstream>

std::ofstream out("books.csv");
out << "title,price,url\n";

for (const auto& book : books) {
    out << '"' << book.title << '"' << ','
        << '"' << book.price << '"' << ','
        << '"' << book.url << '"' << '\n';
}

#include <fstream>

std::ofstream out("books.csv");
out << "title,price,url\n";

for (const auto& book : books) {
    out << '"' << book.title << '"' << ','
        << '"' << book.price << '"' << ','
        << '"' << book.url << '"' << '\n';
}

Option 2: JSON or JSON Lines

Use JSON when records are no longer clean rows. If 1 page has multiple attributes, nested metadata, or optional sections, JSON preserves that structure better than CSV.

A practical C++ choice here is nlohmann/json. JSON is a good fit when:

• Your records have nested fields
• Some pages contain optional keys
• The output needs to feed another application or API

A short example:

#include <nlohmann/json.hpp>
#include <fstream>

nlohmann::json data = nlohmann::json::array();

for (const auto& book : books) {
   data.push_back({
       {"title", book.title},
       {"price", book.price},
       {"url", book.url}
   });
}

std::ofstream out("books.json");
out << data.dump(2);

#include <nlohmann/json.hpp>
#include <fstream>

nlohmann::json data = nlohmann::json::array();

for (const auto& book : books) {
   data.push_back({
       {"title", book.title},
       {"price", book.price},
       {"url", book.url}
   });
}

std::ofstream out("books.json");
out << data.dump(2);

If the crawl is large, JSON Lines is often preferable to a single large JSON array. One record per line is easier to append and easier to recover if the job fails halfway through.

std::ofstream out("books.jsonl");

for (const auto& book : books) {
   nlohmann::json row = {
       {"title", book.title},
       {"price", book.price},
       {"url", book.url}
   };
   out << row.dump() << '\n';
}

std::ofstream out("books.jsonl");

for (const auto& book : books) {
   nlohmann::json row = {
       {"title", book.title},
       {"price", book.price},
       {"url", book.url}
   };
   out << row.dump() << '\n';
}

Option 3: Direct database insert

Write directly to a database when the scraper is part of a larger pipeline, and the output needs to be queried immediately.

For local or single-machine projects, SQLite is the easiest place to start. It makes sense when:

• The scrape feeds an application, not just a file
• You need deduplication or querying
• Multiple runs should write to the same dataset

A minimal SQLite example looks like this:

#include <sqlite3.h>

sqlite3* db = nullptr;
sqlite3_open("books.db", &db);

const char* sql =
   "INSERT INTO books (title, price, url) VALUES (?, ?, ?);";

sqlite3_stmt* stmt = nullptr;
sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr);

for (const auto& book : books) {
   sqlite3_bind_text(stmt, 1, book.title.c_str(), -1, SQLITE_TRANSIENT);
   sqlite3_bind_text(stmt, 2, book.price.c_str(), -1, SQLITE_TRANSIENT);
   sqlite3_bind_text(stmt, 3, book.url.c_str(), -1, SQLITE_TRANSIENT);

   sqlite3_step(stmt);
   sqlite3_reset(stmt);
}

sqlite3_finalize(stmt);
sqlite3_close(db);

#include <sqlite3.h>

sqlite3* db = nullptr;
sqlite3_open("books.db", &db);

const char* sql =
   "INSERT INTO books (title, price, url) VALUES (?, ?, ?);";

sqlite3_stmt* stmt = nullptr;
sqlite3_prepare_v2(db, sql, -1, &stmt, nullptr);

for (const auto& book : books) {
   sqlite3_bind_text(stmt, 1, book.title.c_str(), -1, SQLITE_TRANSIENT);
   sqlite3_bind_text(stmt, 2, book.price.c_str(), -1, SQLITE_TRANSIENT);
   sqlite3_bind_text(stmt, 3, book.url.c_str(), -1, SQLITE_TRANSIENT);

   sqlite3_step(stmt);
   sqlite3_reset(stmt);
}

sqlite3_finalize(stmt);
sqlite3_close(db);

For larger systems, PostgreSQL with libpqxx is the more common next step.

Data-quality patterns worth adding early

Storage becomes easier when records are cleaned before being written.

A few habits pay off quickly:

• Trim whitespace and collapse internal newlines on every extracted string
• Validate types early instead of waiting until insert time
• Log bad records to a side file instead of killing the whole crawl

A small cleaning helper can look like this:

#include <algorithm>
#include <cctype>
#include <regex>

std::string clean_text(std::string value) {
   value = std::regex_replace(value, std::regex(R"(\s+)"), " ");

   value.erase(value.begin(), std::find_if(value.begin(), value.end(),
       [](unsigned char ch) { return !std::isspace(ch); }));

   value.erase(std::find_if(value.rbegin(), value.rend(),
       [](unsigned char ch) { return !std::isspace(ch); }).base(), value.end());

   return value;
}
And a simple side log for bad records can be as small as:
#include <fstream>

std::ofstream bad("bad_records.log", std::ios::app);
bad << "Missing price for title: " << book.title << '\n';

#include <algorithm>
#include <cctype>
#include <regex>

std::string clean_text(std::string value) {
   value = std::regex_replace(value, std::regex(R"(\s+)"), " ");

   value.erase(value.begin(), std::find_if(value.begin(), value.end(),
       [](unsigned char ch) { return !std::isspace(ch); }));

   value.erase(std::find_if(value.rbegin(), value.rend(),
       [](unsigned char ch) { return !std::isspace(ch); }).base(), value.end());

   return value;
}
And a simple side log for bad records can be as small as:
#include <fstream>

std::ofstream bad("bad_records.log", std::ios::app);
bad << "Missing price for title: " << book.title << '\n';

If the scraper starts feeding anything downstream, this is also the right place to think about what data cleaning is. A scraper that stores messy values perfectly is still producing messy data.

The short version is simple: use CSV when the schema is flat, use JSON when the shape is richer, and use a database when the scraper is already part of a real data pipeline.

Advanced C++ web scraping techniques

This is the part where C++ either starts to justify itself or starts to feel expensive. Static page fetching is the easy part. The harder work shows up when the target renders data in the browser, binds sessions to cookies and IPs, needs controlled concurrency, or starts scoring your traffic. C++ can handle all of that, but the engineering cost rises fast, which is why each of the techniques below has a natural escalation point. Tools like Decodo Web Scraping API exist for exactly that handoff.

Handling JavaScript-rendered pages

This is the first hard limit of a static C++ scraper. libcurl and CPR fetch the raw HTTP response. They don't execute page JavaScript, so single-page apps, infinite scroll, and any content added after the initial DOM load often come back as empty containers or placeholders instead of real data. That's not a parser problem. It's a rendering problem.

A quick way to confirm it's to compare the raw HTML against what you see in DevTools. If the class or node you want appears in DevTools but not in the source response, the data is being rendered client-side. At that point, there are really 2 in-house paths. The first is to drive Chrome through the DevTools Protocol from C++, usually over a WebSocket client such as Boost.Beast. That gives you full control, but you also inherit Chromium lifecycle management, browser crashes, and protocol churn. The second is to run a separate headless service, such as Playwright, behind a thin HTTP wrapper and have the C++ binary call it instead. It's less native, but it isolates browser management from the scraper itself. If you need a refresher on the browser side of that stack, this is where what a headless browser is becomes useful. If the target mix is broad or the rendering layer is becoming operational overhead, that's usually when the Decodo Web Scraping API becomes the cleaner production path.

Managing sessions and cookies

Once a target requires a login state or multi-step navigation, plain one-off requests are no longer enough. CPR includes a Session abstraction, which makes it the easiest way to keep request state together in a modern C++ scraper.

If you're using libcurl directly, the classic pattern still works: set both CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR to the same cookie jar path. That lets the scraper read and write cookies across runs. For login-protected targets, log in once, persist the jar, and reuse it. Re-logging on every request adds latency and puts the heaviest pressure on the most heavily protected endpoint. If the site binds a session to 1 IP address, use a sticky proxy so the IP address does not rotate out from under the cookie. That's where Decodo static residential (ISP) proxies fit naturally.

The escalation point here is simple: if you're spending more time maintaining session state, login persistence, and IP affinity than extracting the data itself, stop treating it as “just cookies.” That's an access-layer problem.

Multithreading and connection reuse

This is where C++ starts to earn its keep. Modern C++ lets you scale a scraper with threads, but raw std::thread is usually not the best default anymore. std::jthread in C++20 automatically rejoins on destruction, which makes it safer around early returns and shutdown paths than manual join() bookkeeping.

Even then, more threads aren't the first thing to optimize. In I/O-bound scraping, connection reuse usually matters more than raw thread count. libcurl’s multi interface is designed for multiple simultaneous transfers in a single thread and for scaling transfers into the thousands by keeping a multi-handle over many easy handles. That makes it a better fit for large batches to the same host than spinning up 1 isolated transfer object per request.

A good practical baseline is:

• Start with a small thread count
• Give each worker their own results buffer
• Merge results at the end instead of locking 1 shared vector on every write
• Cap in-flight work so the parser can't fall behind the network indefinitely

The escalation point here is operational, not technical. If you're debugging thread coordination, handle reuse, and queue backpressure more than you're shipping data, the concurrency win may no longer be worth the maintenance cost.

Reducing detection friction

For straightforward targets, a clean request stack is often enough. For stricter ones, you need to make the traffic look less artificial without turning the scraper into a full-time browser-fingerprinting project.

The low-effort wins are still the same:

• Use a realistic browser User-Agent instead of the default curl one
• Send a fuller browser-like header set, not just 1 header
• Add jitter between requests instead of fixed sleeps
• Rotate IPs when repeated requests from 1 address start failing

If you want the broader context behind that, anti-bot systems, web scraping prevention, and how to bypass them are the 2 most useful background reads.

On the library side, libcurl supports proxy configuration directly, while CPR exposes proxy support through its request and session abstractions. If you need rotating identity across a crawl, rotating proxies are the natural fit. If you need a stable identity across a logged-in session, sticky residential or ISP proxies make more sense. For self-managed rotation, that usually means Decodo residential proxies or Decodo rotating proxies.

The deeper problem is fingerprinting. Once you're debugging TLS behavior, CAPTCHA handling, and proxy pool health, you're no longer just tweaking a scraper; you're maintaining a delivery platform. That's the point at which Decodo Web Scraping API or Decodo Site Unblocker becomes easier to justify than another week of in-house unblocking work. If you want to see how deep that rabbit hole goes, check out how to bypass CreepJS and how to bypass CAPTCHAs.

The practical rule

Use C++ to keep the scraper fast, lean, and close to your native pipeline. Don't force it to own every browser, proxy, and unblock problem just because it can. When rendering, cookies, concurrency, and traffic-shaping require more effort than the extraction logic, the cleanest architecture is usually a thinner C++ binary alongside a managed delivery layer, such as the Decodo Web Scraping API.

Limitations and challenges of web scraping in C++

C++ can be excellent for scraping, but it comes with real trade-offs. Some are language-level. Others are scraping problems that become more expensive when the scraper is written in C++.

Language-level limitations

• No first-class scraping framework: There's no C++ equivalent to Scrapy with middleware, item pipelines, retries, and concurrency built in. You assemble the stack yourself.
• No standard headless browser layer: Chrome DevTools Protocol clients exist, but they are thinner and less mature than Playwright or Puppeteer.
• Manual lifetime management: Parser handles, such as htmlDocPtr, require explicit cleanup. If they escape scope without xmlFreeDoc, you leak memory. RAII wrappers help, but they add discipline and complexity to the code.
• Build complexity grows quickly: Every extra library means more CMake configuration, more package manager work, and more ABI risk.

Operational challenges

• Slower iteration: Small selector tweaks still require a rebuild and a rerun. One practical fix is to move XPath strings into config so simple selector changes don't require recompiling.
• Smaller community: There are fewer examples, fewer integrations, and fewer ready-made scraping patterns than you get in Python or Node.
• Harder debugging: Async I/O, thread pools, and handle reuse are all manageable in C++, but debugging them is harder than debugging a Python scraper with a stack trace and a few logs.
• Higher maintenance cost: When the target changes its HTML, the C++ scraper usually needs a rebuild and redeploy, not just a quick script edit.

When the limitations win

A simple rule helps here: if the scraper changes more than once a week, or if it runs at a modest scale, the iteration cost of C++ often outweighs the runtime gain. That's usually the point at which to switch languages, or keep C++ only for the heavy ingestion side of the pipeline. If you want the broader operational view, this is where web scraping at scale is a useful companion read.

C++ web scraping alternatives

C++ isn't the right answer for every scrape. The better question is what you need most right now: speed, browser automation, developer velocity, or simpler deployment. For the full comparison, see the best coding languages for web scraping.

• Python makes more sense when you care more about shipping fast than squeezing every bit of efficiency out of the runtime, which is where the Python web crawler guide could be useful for you.
• JavaScript / Node.js is the better fit when the target is a single-page app and browser automation is central to the job, which is exactly the kind of workflow covered in the JavaScript web scraping guide.
• Rust is worth considering when you want performance on par with C++ but would rather not take on the same memory-management burden; the Rust web scraping guide is the closest comparison.
• Go is usually the better tradeoff when simple concurrency and easy deployment matter more than low-level control, which is why the broader language comparison guide is the best place to weigh that choice.
• C# becomes the natural option when the rest of the system already lives in .NET, and in that case, check out our C# web scraping guide for a good starting point.
• Ruby or PHP is a better fit when scraping is just 1 feature within an existing Rails or Laravel app, and the data volume is moderate, making the Ruby or PHP scraping guide the right next read, depending on your stack.
• If none of those language choices sound worth the maintenance overhead, the cleaner move is to call the Decodo Web Scraping API from whatever language you already use and let it handle rendering, proxies, and unblock logic for you.

Final thoughts

C++ web scraping makes sense when runtime efficiency is the real reason for the project. If you need high throughput, deterministic latency, or tight integration with an existing C++ system, the extra setup can be worth it.

For most projects, the best default stack is still CPR + libxml2, with pugixml as the cleaner option for XML-heavy targets. Add std::jthread or curl_multi when concurrency matters. Wrap parser handles safely. Keep extraction logic simple.

The part C++ does not solve elegantly is everything around delivery: browser rendering, session-heavy targets, proxy rotation, TLS fingerprints, and CAPTCHAs. That's exactly where Decodo Web Scraping API fits best, leaving your C++ scraper as a clean HTTP-in, structured-data-out component.