Python

This Browser-use tutorial shows you how to point an AI agent at a live browser and have it get real work done. Browser-use is the leading open-source library for giving LLM agents browser control – MIT-licensed, with over 110k GitHub stars. By the end, you'll have an agent scraping product data, working through multi-step flows, and handling failures.

A Grubhub promo offer scraper collects the discounts and perks attached to restaurants. Requesting a restaurant page won't return any of them: Grubhub answers with HTTP 200 and 13,631 bytes of JavaScript shell, byte-identical to the homepage. This guide reverse-engineers the 2 Grubhub endpoints that return the data, builds a working promo collector in Python, and measures what each one omits.

HTTPX, Requests, and AIOHTTP all make HTTP requests, but they differ in how they handle concurrency. Requests is synchronous, HTTPX supports sync and async with HTTP/2, and AIOHTTP is async-only and optimized for high concurrency. This article compares their architecture, performance, proxy setup, migration paths, and common production scraping pitfalls.

Proxy rotation sends each request through a different IP, so a site can't link the traffic to one address. Knowing how to rotate proxies in Python helps you avoid issues such as rate limits and IP bans. This guide covers a few ways to do it, from managing your own list to an endpoint that rotates the IP for you.

Wikimon.net is a community-maintained Digimon wiki, but it doesn't offer a public API, so web scraping is the most practical way to pull structured data from its Visual List. In this guide, you’ll build a working Python scraper with Requests and Beautiful Soup to extract Digimon names and image URLs, then save the results to either a CSV file or a MySQL database for later use.

A hidden API is an undocumented endpoint that a website's frontend calls in the background, usually through XHR, Fetch, or GraphQL, to load data as JSON. Scraping hidden APIs means calling that endpoint directly instead of parsing the rendered HTML. Many modern websites load the page first, then fetch the actual data this way, so the clean JSON you need is often just one request away. In this guide, you'll learn how to find these hidden APIs and use them to collect structured data faster, without a headless browser.

Perplexity is slowly becoming one of the more prominent LLMs in 2026 because it delivers output in a unique way that sets it apart from the rest. When you send a prompt, Perplexity AI will return a direct answer, along with cited sources (URLs and titles), related follow-up questions to that prompt, and supporting images as well. All this information can be worth tracking for businesses that care about their brand visibility, and how Perplexity is representing them and their competitors to a typical user.

Python and JavaScript are 2 languages that dominate web scraping, but for different reasons. The real question isn't which language is "better," but rather what task you're building for. This article compares both languages in terms of libraries, performance, support for dynamic content, and anti-bot strategies, while also showing why the overall architecture matters more than your language choice.

When you need to get cookies in Playwright and reuse them across runs, the key concept is the browser context. Cookies live in the context, not the page. Getting cookies in Playwright starts with the context.cookies() method, and returns every cookie stored in the browser context, which you can save and load from a file later. This guide walks through how to get, save, and load cookies in Playwright.

Knowing how to wait for a page to load in Playwright is the difference between a scraper that returns clean data and one that fails silently. In this guide, you'll learn how to handle waiting in Playwright, including how it behaves in a headless browser environment, covering auto-waiting, selectors, network events, timeouts, custom conditions, and error handling across dynamic pages.

Undetected ChromeDriver is a Python library that patches Selenium’s ChromeDriver to avoid bot detection when web scraping. Standard Selenium ChromeDriver is blocked by most protected websites within the first few requests: anti-bot services like Cloudflare, DataDome, and HUMAN (formerly PerimeterX) read automation flags, WebDriver properties, and browser-fingerprint gaps before the first page finishes loading. The undetected_chromedriver library works as a drop-in Selenium WebDriver replacement (swap webdriver.Chrome() for uc.Chrome()) and reduces those signals. But it does not hide your IP address, so this guide also shows how to pair it with residential proxies and behavioral techniques to stay unblocked.

Bing scraping is the automated extraction of rankings, ads, snippets, and search features from Bing's SERPs. Since Microsoft retired all official Bing Search APIs in 2025, scraping and third-party SERP APIs are the main ways to access this data programmatically. This guide covers Python-based approaches using Requests, Beautiful Soup, Playwright, and Decodo's Web Scraping API.

JSON is the format that most web APIs and modern websites use to send their data. This tutorial shows how to scrape JSON data in Python – fetching it, parsing it, modifying it, and exporting clean files. You'll also learn about the tools for messy or oversized responses, and how to get data when sites block you with fingerprinting.

Scrapy is an asynchronous Python framework for crawling and extracting data at scale, but it doesn't execute JavaScript on its own. A spider can get a clean 200 response and still return empty selectors on a modern site. This guide covers the rendering options (Splash, Selenium, Playwright, managed APIs) and the cache and concurrency settings that matter once browser rendering comes into play.

When you need to scrape 50 pages of search results behind a login wall, raw Requests + Beautiful Soup force you to track cookies and assemble form payloads by hand, while Selenium launches a full browser for pages that don't even use JavaScript. MechanicalSoup sits between those extremes. It wraps Requests and Beautiful Soup into a stateful browser that handles web scraping sessions, forms, and navigation automatically. This guide covers everything from installation to proxy-powered production scrapers.

A news crawler is a tool that automatically pulls content from news websites. A web news crawler helps with tracking competitors, feeding LLM pipelines, or watching topic coverage across publishers. This guide walks you through building a configurable proxy-integrated Python news crawler that’ll target multiple news sources, handles proxy rotation, and saves structured results on a schedule.

Extracting text from HTML in Python is one of the most common tasks in web scraping, NLP pipelines, search indexing, and data preparation. The goal is to keep the visible content from a webpage while removing all the HTML markup, scripts, and styles that surround it. This guide walks you through the popular Python libraries for HTML text extraction and a full step-by-step workflow to go from raw HTML to clean, production-ready text.

On a recent run, wired.com ranked no. 1 on US desktop and no. 2 on UK desktop computers for "best laptop 2026". Same query, same hour, different country. That gap is what a single-number rank tracker misses, especially now that modern SERPs add AI Overview citations, featured snippets, and "People also ask" blocks that most older tracking tools ignore. This post walks through building a tracker that captures all of it, starting with a manual baseline for ground truth, then moving to a Python implementation against a SERP API, and finally setting up a scaling path for more keywords and locations.