Data Collection

Puppeteer is a Node.js library that controls headless Chromium for browser automation and web scraping. Without proxies, every request goes out from your real IP, and on protected sites, that IP gets blocked fast. This guide covers every method for configuring, authenticating, and rotating proxies in Puppeteer, plus how to troubleshoot the failures you'll actually run into. Not familiar with how proxies work? Check out what a proxy server is before continuing.

Scala web scraping fits naturally into JVM data pipelines, sharing types and libraries with your Spark, Akka, and Kafka code. This guide covers everything you need to ship a production scraper: environment setup, library selection, pagination, JavaScript rendering, anti-bot mitigation, and structured data export. Including when a managed scraping API is the smarter call.

If you're choosing Hermes Agent vs. OpenClaw, you're looking at the two most popular open-source agent frameworks of the year. Both run AI agents on their own, work inside your messaging apps, and call tools for you. This guide compares both of these agents' features, use cases, and possible third-party integrations.

Web scraping with Node Fetch offers a lightweight way to collect data in Node.js. By fetching raw HTML or JSON responses and pairing them with parsers like Cheerio, developers can transform unstructured pages into structured datasets. This Node Fetch tutorial explains request handling, response parsing, data extraction, proxy integration, and when managed scraping APIs are necessary to effectively bypass advanced anti-bot protections.

Watir is an open-source Ruby library for automating web browsers through code. Built on top of Selenium WebDriver, it wraps browser communication in a clean, Ruby-idiomatic API so you can focus on clicking buttons, filling forms, navigating pages, and extracting data without managing the underlying complexity. It's particularly useful for scraping JavaScript-heavy sites, automating form submissions, and collecting content that only appears after user interaction. This guide walks you through the full process, from setup to a working Watir scraper with proxy support.

A residential proxy network routes your traffic through real ISP-assigned home IPs. But how routing actually happens, which proxy types you end up using, and how IPs are sourced have a much bigger impact on success rates. This guide breaks down how residential proxy networks work, including ASN-level routing, the different types of IPs, and the protocols they support.

Thanks to blending Java’s massive ecosystem with a scripting-friendly syntax, Groovy works as a practical alternative for web scraping on the JVM. This guide shows you how to scrape websites with the HTTP Jodd client, parse HTML documents, manage sessions, utilize Jodd Lagarto and Jerry, and use Selenium to automate browsers. You'll also learn how to configure proxies for real-world, block-resistant scraping.

Web scraping with Claude Fable 5 turns slow data collection into a fast, mostly hands-off job. Fable 5 is Anthropic's most capable model (yet), and it can write scrapers, run them, repair its own errors, and return clean, structured data. This article walks through setup, real use cases, prompt patterns, costs, and the limits worth knowing.

No-code web scraping with n8n lets you build automated scrapers on a visual canvas – no Python, no server, no terminal. You need prices tracked weekly or listings monitored daily, but can't code a scraper or run one. This guide shows you how to use only drag-and-drop nodes that run on schedule in the cloud.

Golang Colly is a fast, callback-driven scraping framework for the Go programming language. It wraps HTTP requests, HTML parsing, rate limiting, and concurrency in a clean API, so you can pull structured data from a website with very little code. This tutorial walks you through building a working Colly scraper from an empty project all the way to proxy rotation.

Vibe scraping is the practice of building scrapers by describing goals in natural language to an LLM rather than hand-writing selectors, a concept derived from Andrej Karpathy's 'vibe coding.' This allows developers to turn prompts into working extractors as LLMs now efficiently parse DOMs, infer schemas, and write code. While it enables rapid prototyping, it introduces new failure modes like hallucinated selectors; scaling these scripts for production still requires real proxies and rendering infrastructure.

Puppeteer makes browser automation feel easy until you need to save a file to disk. Triggering a download in headless mode isn't the same as clicking a button in a real browser, and the default behavior in headless Chrome won't help you. This guide covers the full Puppeteer download file workflow: configuring CDP correctly, picking the right method for your scenario, detecting when a file truly finished, and scaling to batch jobs without leaking memory or corrupting your queue.

Price data is important for monitoring competitors in eCommerce, enforcing MAP policies, and receiving deal alerts. Doing this manually isn't effective for scaling. A practical approach is price scraping, which helps automatically collect product pricing data from eCommerce websites. This guide will show you how to build a Python scraper using Playwright. It will help you gather real prices, deal with anti-bot measures, and create structured JSON data.

C++ web scraping is the process of sending HTTP requests from a C++ program, retrieving HTML or other structured responses, and parsing the data using libraries such as libcurl, CPR, libxml2, or pugixml. It's most useful in scraping workloads where CPU efficiency, memory control, predictable latency, or direct integration with an existing C++ system matter more than quick setup. That makes it a practical option for performance-critical pipelines, but a heavier one to build and maintain. The real question isn't whether C++ can scrape the web. It's whether that extra control is worth the extra engineering work.

Submitting forms with Puppeteer goes beyond just clicking a button on the browser or typing text into an input box. Puppeteer is a Node.js library that can control a headless or headful Chromium (browser) instance through the DevTools Protocol. That means you can use Puppeteer to automatically locate form fields, fill input boxes with necessary values, trigger the action of submitting a form, and confirm whether that form submission actually worked. If you’re into web scraping, or you’re testing your own product, or even automating anything in a browser with JavaScript, you’ll inevitably run into form fields and submit buttons that you will need to get passed programmatically with Puppeteer.

When you need to scrape 50 pages of search results behind a login wall, raw Requests + Beautiful Soup force you to track cookies and assemble form payloads by hand, while Selenium launches a full browser for pages that don't even use JavaScript. MechanicalSoup sits between those extremes. It wraps Requests and Beautiful Soup into a stateful browser that handles web scraping sessions, forms, and navigation automatically. This guide covers everything from installation to proxy-powered production scrapers.

Cloudflare is a massive global cloud network that sits firmly between your scraper and the data you need, blocking all requests that fail its multi-layered detection system. It powers nearly 21% of all websites globally, meaning that 1-in-5 sites rely on this network. Therefore, knowing how to bypass it is essential for serious scrapers. This practical walkthrough covers detection methods, tools like Puppeteer and Playwright, and both DIY approaches and managed solutions, including proxy strategies and web scraping APIs.

If you try to run curl on an infinite scroll page and then search (grep) for the content, the result will show zero matches. The required items aren't available in the initial HTML. The content is loaded using fetch requests that the page sends when a scroll event is triggered. This guide will cover: diagnosing the target, three scroll strategies, verification, speed, anti-detection methods, and a real-world walkthrough.