Data Collection

If you work with APIs, webhooks, automation scripts, or web scraping, chances are you've needed to send JSON with cURL. cURL is one of the most widely used command line tools for making HTTP requests, and modern APIs almost always rely on JSON payloads. In this guide, you’ll learn how to send JSON with cURL, work with files and authentication, debug requests, and route traffic through proxies when needed.

Elixir web scraping solves one of the hardest problems in high-volume data collection: concurrency without thread overhead. The BEAM virtual machine (Erlang's runtime) runs each HTTP request as a lightweight process, not an OS thread, so you can fetch thousands of pages concurrently. If a process crashes, the supervisor restarts it automatically. This guide builds a complete Elixir scraper from scratch, covering static pages, paginated targets, JavaScript-heavy sites, and anti-bot countermeasures.

A scraper that fetches pages one at a time spends most of its time waiting on the network. Asynchronous web scraping in Python (built on asyncio and aiohttp) fixes that by handling many requests at once on a single event loop. This guide walks through building a working async scraper, then covers the proxy, retry, and anti-bot escalation patterns you'll need at scale.

​​HTML structure shifts constantly, but the visible text on a page tends to remain more stable. That stability is what makes text-based selectors useful in web scraping. This guide covers the core functions you need to work with text in XPath: text(), contains(), starts-with(), normalize-space(), and translate(), including where each one breaks and how to combine them to build selectors that survive page updates.

If you're building a Playwright scraper and not using XPath, you're probably leaving your most precise location strategy on the table. Think of the DOM as a tree of nodes, and an XPath expression as the specific zip code to reach any node. In this article, we'll explain Playwright XPath fundamentals, how to construct expressions, and how to interact with elements, including real-world examples.

Every JSON payload a JavaScript app receives passes through JSON.parse(). The method has to handle malformed payloads, lossy type round-trips, files that exceed Node's default heap, and APIs that change shape without warning. Each of those failures surfaces at parse time. This guide covers them all, including practical patterns for web scraping.

Puppeteer and Selenium are the 2 most-used browser automation tools for scraping JavaScript-heavy pages. Comparing puppeteer vs selenium for web scraping isn't just about speed. Browser support, language support, and anti-bot handling all play a role. This guide covers what each tool does well, what it doesn't, and how to pick the right one.

A mobile proxy is an intermediary that routes your traffic through a cellular network. This alone sets them apart from other proxy types in terms of anti-detection, geo-targeting granularity, and content exclusivity. This guide will cover how mobile proxies work under the hood, how they stack up against residential, ISP, and datacenter alternatives, and how you can configure one for yourself.

Most developers already know jQuery for DOM manipulation – it's been the default "make the page do things" library for over a decade. So when you need to scrape some data from a web page, reaching for $('.price').text() feels instinctive. The catch is that jQuery web scraping works differently depending on where you run it. In the browser, CORS will shut you down fast. In Node.js, you need a simulated DOM before jQuery even loads. This guide covers both paths – selectors, $.get(), pagination, server-side setup with jsdom, and when to ditch jQuery for something built for the job.

cURL Basic Auth takes 30 seconds until the password has a $ in it, or a colon, or the server keeps returning 401. This guide covers every syntax variation, how to build the Authorization header manually, handling special characters, keeping credentials out of shell history and CI/CD scripts, and when Basic Auth is the wrong tool entirely.

PowerShell is already where many Windows admins, DevOps teams, and automation-minded developers handle repetitive work. That makes web scraping a natural next step when you need product prices, uptime signals, public data for reports, or quick checks from the terminal. PowerShell works well here because output is pipeable, objects are native, and CSV and JSON exports are built in. In this guide, you'll build a scraper that fetches pages, parses HTML, handles pagination and errors, uses proxies when needed, and exports structured data.

Python has the richest scraping ecosystem of any language. That breadth is exactly why making a choice is harder than it should be. This article continues from our Python web scraping guide, focusing on the selection problem: 8 libraries across 4 categories, what each one does best, where it breaks down, and how to choose the right one for the job.

Web scraping used to mean extracting data with CSS selectors, and then rebuilding your scraper every time a target changes its layout. Here's the good news: ScrapeGraph AI takes a new approach as it uses LLMs to extract data from websites based on meaning, so you can describe what you need in natural language while the library handles the rest for you. In this guide, you'll learn how ScrapeGraph AI works and how to configure it to export structured datasets in the right formats. The tools we'll use are Python, ScrapeGraph AI, and Decodo proxies.

A plain Go HTTP client only sees the HTML the server returns. That's enough for static pages. It breaks down when JavaScript renders the real content later, which is common on SPAs, infinite-scroll interfaces, and login-protected flows. chromedp solves that by driving Chrome or Chromium through the Chrome DevTools Protocol, or CDP, without a separate WebDriver layer. In this tutorial, you’ll set up chromedp, extract dynamic content, interact with pages, route traffic through proxies, run Chrome in Docker, and scale scraping with goroutines.

Java is a battle-tested choice for web scraping at scale due to its robust type safety, structured concurrency, safe multithreading, and a mature ecosystem. However, its advantage is also a major pain point: having too many libraries. From jsoup and HtmlUnit to Selenium and Playwright, these libraries exist to simplify web scraping, and yet picking "the right one" is a challenge. This guide will teach you how to choose the right tool based on your project requirements and how to handle modern scraping challenges.

Waiting for a page to load when using Beautiful Soup is a common challenge in web scraping, especially when your scraper returns empty results because the page renders content via JavaScript. This happens because Beautiful Soup is a parser, not a browser, so it can’t execute JavaScript or wait for dynamic content to load. To handle this, you can use browser automation tools like Selenium or Playwright, a lightweight option like requests-html, or a Web Data API for production-grade workflows.

Scraping a page is simple. Crawling an entire website repeatedly, at scale, while also producing structured data that you can query, can be complex. Most scraping tools aren't designed for it, and that's what Apache Nutch is developed for. Nutch is an open source web crawler with built-in robots.txt compliance and native Apache Solr integration. By the end of this guide, you'll have a scoped crawl pipeline running and your data indexed into Solr.

Web scraping without getting blocked is one of the hardest challenges you might face. Whether you’re a business conducting market research or a solopreneur working on your next big thing, most scrapers fail not because the code is wrong, but because websites now run layered detection that flags bots before a single byte of HTML is returned. This guide breaks down all the detection layers, including network, TLS, browser, and behavioral, and delivers the best techniques on how to overcome each.