Data Collection

A SOCKS proxy is an internet protocol that routes network traffic through a third-party server and masks your IP address. It operates at the transport layer, handling any traffic type – TCP, UDP, web, gaming, torrenting, and more – without inspecting or modifying the data. In this blog entry, we’ll cover the ins and outs of SOCKS proxies, including how they work, key benefits, use cases, and how they compare against HTTP proxies.

Web scraping with Linux is more capable than most people expect. Bash may not be the go-to tool for web scraping, but with a handful of pre-installed command-line utilities you can build a working scraper without touching Python or a browser. This guide covers how to make HTTP requests in Linux, parse into HTML and JSON, set up proxy support with Decodo, and build a fully working Bash-based scraper from scratch.

Perplexity is slowly becoming one of the more prominent LLMs in 2026 because it delivers output in a unique way that sets it apart from the rest. When you send a prompt, Perplexity AI will return a direct answer, along with cited sources (URLs and titles), related follow-up questions to that prompt, and supporting images as well. All this information can be worth tracking for businesses that care about their brand visibility, and how Perplexity is representing them and their competitors to a typical user.

Python and JavaScript are 2 languages that dominate web scraping, but for different reasons. The real question isn't which language is "better," but rather what task you're building for. This article compares both languages in terms of libraries, performance, support for dynamic content, and anti-bot strategies, while also showing why the overall architecture matters more than your language choice.

When you need to get cookies in Playwright and reuse them across runs, the key concept is the browser context. Cookies live in the context, not the page. Getting cookies in Playwright starts with the context.cookies() method, and returns every cookie stored in the browser context, which you can save and load from a file later. This guide walks through how to get, save, and load cookies in Playwright.

Headless browsers in Playwright can leak fingerprint signals that anti-bot systems notice. While Playwright is great for automation, its default settings make these signals easy to spot. Stealth plugins help cover these leaks so your scripts look like real user traffic. This guide explains detection methods, how to set up stealth in Python and Node.js, what gets patched, how to test, and the limits and scaling options.

Knowing how to wait for a page to load in Playwright is the difference between a scraper that returns clean data and one that fails silently. In this guide, you'll learn how to handle waiting in Playwright, including how it behaves in a headless browser environment, covering auto-waiting, selectors, network events, timeouts, custom conditions, and error handling across dynamic pages.

Undetected ChromeDriver is a Python library that patches Selenium’s ChromeDriver to avoid bot detection when web scraping. Standard Selenium ChromeDriver is blocked by most protected websites within the first few requests: anti-bot services like Cloudflare, DataDome, and HUMAN (formerly PerimeterX) read automation flags, WebDriver properties, and browser-fingerprint gaps before the first page finishes loading. The undetected_chromedriver library works as a drop-in Selenium WebDriver replacement (swap webdriver.Chrome() for uc.Chrome()) and reduces those signals. But it does not hide your IP address, so this guide also shows how to pair it with residential proxies and behavioral techniques to stay unblocked.

Bing scraping is the automated extraction of rankings, ads, snippets, and search features from Bing's SERPs. Since Microsoft retired all official Bing Search APIs in 2025, scraping and third-party SERP APIs are the main ways to access this data programmatically. This guide covers Python-based approaches using Requests, Beautiful Soup, Playwright, and Decodo's Web Scraping API.

JSON is the format that most web APIs and modern websites use to send their data. This tutorial shows how to scrape JSON data in Python – fetching it, parsing it, modifying it, and exporting clean files. You'll also learn about the tools for messy or oversized responses, and how to get data when sites block you with fingerprinting.

Scrapy is an asynchronous Python framework for crawling and extracting data at scale, but it doesn't execute JavaScript on its own. A spider can get a clean 200 response and still return empty selectors on a modern site. This guide covers the rendering options (Splash, Selenium, Playwright, managed APIs) and the cache and concurrency settings that matter once browser rendering comes into play.

As a Ruby developer, you must have used proxies for multiple applications, including web scraping, API integration, and geo-targeted testing. Without a proxy, every request leaves from the same IP, which is the fastest way to get rate-limited or blocked. In this guide, you'll learn how to configure a Ruby proxy with Net::HTTP and Faraday, add authentication, rotate IPs, and connect Ruby applications to Decodo residential proxies.

Pyppeteer is an unofficial Python port of Puppeteer, the Node.js library that drives headless Chromium through the DevTools Protocol. It brings the same async model to Python for clicking, filling forms, waiting, and scraping JavaScript-heavy sites. It works, but it's no longer the 2026 default. This guide covers using it and when to switch to Playwright or nodriver.

Building reliable web scrapers can get complex and difficult to maintain, but Crawlee aims to simplify the process. As project needs grow, developers often encounter challenges that require multiple tools and configurations. Crawlee eliminates the need to build these configurations from scratch or migrate to a different tool mid-crawl, allowing you to focus on your scraping logic instead. In this guide, you'll learn how to scrape using Crawlee's 3 main crawler classes. We'll also explore the routing architecture, proxy integration with Decodo, and data storage.

Most Python scrapers that use Requests stop working as soon as a site is protected by Cloudflare. You might see a 403 error, get stuck in a redirect loop, or land on a "Just a moment..." page that never loads. Cloudscraper solves this problem without needing a headless browser. It builds on Requests, handles Cloudflare's JavaScript challenges, and gives you a working session. This guide explains how to set up Cloudscraper, configure proxies, choose an interpreter, handle CAPTCHAs, parse data, fix common errors, and understand the library's limitations. If you're new to Python scraping, start with the Python web scraping guide first.

When you scrape the web with Puppeteer, you almost always pull in data you want alongside extras you don't need, like images, fonts, and tracking scripts that increase your request count, slow your pages, and drain your proxy bandwidth. In this guide, you'll learn how to block unnecessary requests with request interception and Chrome DevTools Protocol (CDP) so your scraper runs faster and scales more efficiently.

Kotlin developers don't need to reach for Python to scrape. The JVM ecosystem covers the full stack: Jsoup for HTML parsing, OkHttp for HTTP requests, and coroutines for concurrency. This guide is for JVM and Android developers, as well as Java teams evaluating a migration. By the end of this piece, you'll have a working scraper that handles pagination, runs concurrent requests, integrates proxies, and exports data to CSV.

Yes, you can parse HTML with regex – but only for specific tasks. Regex works well on flat targets like meta tags, sitemap URLs, or inline JSON-LD. But on nested or JavaScript-rendered markup, it fails silently, and you often don’t notice until the data is already wrong. This guide explains when regex works on HTML and when it breaks, includes working Python for the common extraction tasks (meta tags, JSON-LD, bulk extraction), and covers when to switch to a parser or get past a page that blocks you.