Java Web Scraping Libraries: How to Choose and Use the Best Tools for Your Project

TL;DR

• Java is a popular choice for web scraping due to its cross-platform portability, built-in multithreading, type safety, and mature ecosystem.
• Choose a library based on the type and size of your target content, the amount of control you need and maintenance you can do, and anti-bot resistance level.
• jsoup, HtmlUnit, Gecco, and HttpClient + jsoup are suitable for single-threaded scripts.
• Apache Nutch, Selenium, WebMagic, Gecco, and Playwright for Java are good choices for dynamic content.
• Use rotating residential proxies to scale scraping and avoid common issues.
• Use Web Scraping API to manage parsing, browser rendering, anti-bot bypass, and proxy rotation.

Why use Java for web scraping?

Among web scraping languages, Java is an excellent choice for large-scale web scraping because it provides users with a full scraping package: HttpClient to fetch, jsoup to parse, and Selenium and Playwright for dynamic pages. It’s also highly useful for enterprise systems, non-stop crawlers, and existing Java infrastructure. These are key features that make web scraping with Java so strong:

• Built-in multithreading for concurrent scraping (java.util.concurrent). Virtual Threads allow high-volume tasks to run in parallel without exhausting system resources.
• Type safety allows consistency and data integrity by preventing runtime errors typical for large-scale data extraction.
• The high-performance Java Virtual Machine (JVM) has the "Write Once, Run Anywhere" (WORA) capability, meaning it can be run on any server environment, therefore providing cross-platform portability.
• Mature ecosystem with a robust set of libraries, HTTP clients, and HTML parsers.
• Compiled applications can manage large volumes of data faster than interpreted languages.
• Long-term maintainability makes it ideal for long-running projects, particularly for enterprises.  

Like any language, this one comes with some tradeoffs as well. Compared to Python or Node.js, Java needs more code for the same task, but its reliability at scale justifies its usage. Also, it’s not suitable for small and fast projects due to lower speeds.

How to choose the right Java web scraping library?

Choosing a library seems like a daunting task, but this guide aims to help you find the best-suited one for your specific project. Before exploring each individual library, let’s go through the questions that will help you select the right tool. 

Question 1: Is the content static or dynamic? 

Use jsoup for static HTML, but Selenium and Playwright are better suited for the JavaScript (JS)-heavy target sites.

Question 2: Does your project prioritize speed or fidelity? 

Lightweight parsers (e.g. Beautiful Soup) are much faster and resource-efficient than headless browsers (e.g. Selenium), because the latter option needs time to load and render page content. However, headless browsers are crucial and more accurate for JS-heavy apps and complex pages.

Question 3: What are your project’s scale requirements? 

Choose between a single-threaded script and a distributed crawler. A single-threaded script is a simple, low-resource tool for crawling small websites consecutively and quickly on a single machine. A distributed crawler is complex and used for concurrent scraping of massive datasets, providing speed, fault tolerance, and real-time data. 

However, different libraries suit different scales: lightweight libraries like jsoup and Crawler4j are ideal for simple HTML parsing and crawling (single-threaded scripting), while robust frameworks such as Apache Nutch are able to manage massive data and horizontal scaling.

• Java libraries for single-threaded scripts include: jsoup, HtmlUnit, Gecco, and HttpClient + jsoup.
• Java frameworks for distributed crawlers include: Apache Nutch, Selenium, WebMagic, Gecco, and Playwright for Java.
• A hybrid commonly used inside distributed systems: HttpClient + jsoup

Question 4: How much maintenance are you able/willing to commit to? 

Make sure to evaluate a library's GitHub activity, release frequency, and community size and activity before committing to it, as this will ensure long-term maintenance, security, and stability. You want to avoid abandoned tools. Here’s a general ranking of popular libraries by the frequency of maintenance needed:

• Highest maintenance need: Selenium, HTMLUnit
• High: Gecco
• Medium-high: WebMagic
• Medium: Jauntium, Apache Nutch
• Medium-low: Playwright (Java)
• Low: HttpClient + jsoup
• Very low: jsoup

Question 5: What anti-bot resistance level do you need? 

No matter which Java library you end up choosing, you’ll need proxies and/or a web scraping API for JavaScript rendering and to avoid related challenges. Proxies will allow you to counteract the target site’s anti-bot protections, IP bans, rate limits, geo-restrictions, browser fingerprinting, and CAPTCHAs.

Use web scraping APIs for sites with strong anti-bot technology and for browser rendering. Otherwise, you may end up spending more time on infrastructure than on actual scraping. If you need a fast, all-in-one, "skip-the-complexity" solution that manages parsing, browser rendering, anti-bot bypass, and proxy rotation, use Decodo's Web Scraping API. Alternatively, you can utilize Decodo’s Site Unblocker for protected sites.

A decision diagram:

Best Java web scraping libraries compared

This summary table provides a comparison of all libraries covered in the article for quick reference.

jsoup: lightweight HTML parsing

jsoup is the go-to library for parsing static HTML. Via an intuitive and user-friendly API, it simplifies extracting data from URLs, files, or strings; fetches HTML and parses it using DOM traversal, CSS, and XPath selectors. jsoup implements the WHATWG HTML5 specification and parses a variety of HTML to the same DOM as modern browsers. 

Strengths: Simple API, CSS selectors, DOM manipulation, and small footprint. It’s self-contained with no external dependencies.

Limitations: As a pure HTML parser, jsoup doesn’t execute JavaScript; it’s ineffective for SPAs or pages with dynamically loaded content.

When to use: Parsing, extracting, or manipulating data from HTML or XML documents, static HTML pages with no complex browser interactions, RSS/XML feeds.

Code example: Fetching a page and extracting product titles using CSS selectors with Decodo proxy integration.

Download the jsoup dependency.

curl -O https://jsoup.org/packages/jsoup-1.17.2.jar

curl -O https://jsoup.org/packages/jsoup-1.17.2.jar

Save the below file as ScrapeBooksWithProxy.java and replace YOUR_PROXY_USERNAME and YOUR_PROXY_PASSWORD with your credentials.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;




import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.Proxy;
import java.net.InetSocketAddress;




public class ScrapeBooksWithProxy {
    public static void main(String[] args) {
        try {
            // Proxy credentials (replace with your Decodo details)
            String proxyHost = "gate.decodo.com";
            int proxyPort = 7000;
            String proxyUser = "YOUR_PROXY_USERNAME";
            String proxyPass = "YOUR_PROXY_PASSWORD";




            // Set global authenticator for proxy auth
            Authenticator.setDefault(new Authenticator() {
                @Override
                protected PasswordAuthentication getPasswordAuthentication() {
                    return new PasswordAuthentication(
                        proxyUser,
                        proxyPass.toCharArray()
                    );
                }
            });




            // Create proxy
            Proxy proxy = new Proxy(
                Proxy.Type.HTTP,
                new InetSocketAddress(proxyHost, proxyPort)
            );




            // Fetch the page through proxy
            String url = "http://books.toscrape.com/";
            Document doc = Jsoup.connect(url)
                                .proxy(proxy)
                                .userAgent("Mozilla/5.0")
                                .timeout(15_000)
                                .get();




            // Select titles using CSS selector, extract (each title is inside: <h3><a title="...">)
            Elements titles = doc.select("article.product_pod h3 a");




            for (Element el : titles) {
                String title = el.attr("title");
                System.out.println(title);
            }




        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;




import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.Proxy;
import java.net.InetSocketAddress;




public class ScrapeBooksWithProxy {
    public static void main(String[] args) {
        try {
            // Proxy credentials (replace with your Decodo details)
            String proxyHost = "gate.decodo.com";
            int proxyPort = 7000;
            String proxyUser = "YOUR_PROXY_USERNAME";
            String proxyPass = "YOUR_PROXY_PASSWORD";




            // Set global authenticator for proxy auth
            Authenticator.setDefault(new Authenticator() {
                @Override
                protected PasswordAuthentication getPasswordAuthentication() {
                    return new PasswordAuthentication(
                        proxyUser,
                        proxyPass.toCharArray()
                    );
                }
            });




            // Create proxy
            Proxy proxy = new Proxy(
                Proxy.Type.HTTP,
                new InetSocketAddress(proxyHost, proxyPort)
            );




            // Fetch the page through proxy
            String url = "http://books.toscrape.com/";
            Document doc = Jsoup.connect(url)
                                .proxy(proxy)
                                .userAgent("Mozilla/5.0")
                                .timeout(15_000)
                                .get();




            // Select titles using CSS selector, extract (each title is inside: <h3><a title="...">)
            Elements titles = doc.select("article.product_pod h3 a");




            for (Element el : titles) {
                String title = el.attr("title");
                System.out.println(title);
            }




        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

From the same directory, compile and run.

javac -cp ./jsoup-1.17.2.jar ScrapeBooksWithProxy.java
java -cp .:./jsoup-1.17.2.jar ScrapeBooksWithProxy

javac -cp ./jsoup-1.17.2.jar ScrapeBooksWithProxy.java
java -cp .:./jsoup-1.17.2.jar ScrapeBooksWithProxy

HtmlUnit: headless browser with JavaScript support

HtmlUnit is a powerful "GUI-less" library that acts as a middle-ground option between static parsers like jsoup and heavy browser automation tools like Selenium. It is a headless browser that simulates a browser in Java, executes JavaScript without needing an external browser binary, and handles cookies and sessions. It allows browser interaction: clicking, form submissions, and navigating through pages. 

Strengths: Integrates natively with Java projects, handles most JavaScript, supports form submission and page navigation, no external browser dependencies, and can emulate different browsers (Chrome, Firefox, Edge).

Limitations: HtmlUnit’s custom JavaScript engine (Rhino/Nashorn) struggles with modern frameworks (React, Vue, Angular). It’s prone to errors when encountering modern CSS or non-standard HTML and can trigger anti-bot detection.

When to use: Scraping sites with moderate JavaScript, automated testing, form submission, and cookie handling. 

Code example: Navigating to a page, waiting for JavaScript to execute, and extracting rendered content with Decodo proxy integration.

To run this example, first download HtmlUnit and its dependencies:

curl -L -o htmlunit.zip https://repo1.maven.org/maven2/org/htmlunit/htmlunit/4.8.0/htmlunit-4.8.0-bin.zip && unzip htmlunit.zip

curl -L -o htmlunit.zip https://repo1.maven.org/maven2/org/htmlunit/htmlunit/4.8.0/htmlunit-4.8.0-bin.zip && unzip htmlunit.zip

This will create a folder containing all required .jar files inside a lib directory.

Next, save the following code as HtmlUnitScraperWithProxy.java:

import org.htmlunit.WebClient;
import org.htmlunit.WebClientOptions;
import org.htmlunit.ProxyConfig;
import org.htmlunit.html.HtmlPage;
import org.htmlunit.html.DomElement;


import java.net.Authenticator;
import java.net.PasswordAuthentication;


public class HtmlUnitScraperWithProxy {
    public static void main(String[] args) {


        String proxyHost = "gate.decodo.com";
        int proxyPort = 7000;
        String proxyUser = "YOUR_PROXY_USERNAME";
        String proxyPass = "YOUR_PROXY_PASSWORD";


        Authenticator.setDefault(new Authenticator() {
            @Override
            protected PasswordAuthentication getPasswordAuthentication() {
                return new PasswordAuthentication(proxyUser, proxyPass.toCharArray());
            }
        });


        try (final WebClient webClient = new WebClient()) {


            WebClientOptions options = webClient.getOptions();


            options.setJavaScriptEnabled(true);
            options.setCssEnabled(false);
            options.setThrowExceptionOnScriptError(false);


            ProxyConfig proxyConfig = new ProxyConfig(proxyHost, proxyPort);
            options.setProxyConfig(proxyConfig);


            String url = "http://quotes.toscrape.com/js/";
            HtmlPage page = webClient.getPage(url);


            webClient.waitForBackgroundJavaScript(5000);


            for (DomElement quote : page.querySelectorAll(".quote")) {
                DomElement textEl = quote.querySelector(".text");
                DomElement authorEl = quote.querySelector(".author");


                if (textEl != null && authorEl != null) {
                    String text = textEl.asNormalizedText();
                    String author = authorEl.asNormalizedText();


                    System.out.println(text + " — " + author);
                }
            }


        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

import org.htmlunit.WebClient;
import org.htmlunit.WebClientOptions;
import org.htmlunit.ProxyConfig;
import org.htmlunit.html.HtmlPage;
import org.htmlunit.html.DomElement;


import java.net.Authenticator;
import java.net.PasswordAuthentication;


public class HtmlUnitScraperWithProxy {
    public static void main(String[] args) {


        String proxyHost = "gate.decodo.com";
        int proxyPort = 7000;
        String proxyUser = "YOUR_PROXY_USERNAME";
        String proxyPass = "YOUR_PROXY_PASSWORD";


        Authenticator.setDefault(new Authenticator() {
            @Override
            protected PasswordAuthentication getPasswordAuthentication() {
                return new PasswordAuthentication(proxyUser, proxyPass.toCharArray());
            }
        });


        try (final WebClient webClient = new WebClient()) {


            WebClientOptions options = webClient.getOptions();


            options.setJavaScriptEnabled(true);
            options.setCssEnabled(false);
            options.setThrowExceptionOnScriptError(false);


            ProxyConfig proxyConfig = new ProxyConfig(proxyHost, proxyPort);
            options.setProxyConfig(proxyConfig);


            String url = "http://quotes.toscrape.com/js/";
            HtmlPage page = webClient.getPage(url);


            webClient.waitForBackgroundJavaScript(5000);


            for (DomElement quote : page.querySelectorAll(".quote")) {
                DomElement textEl = quote.querySelector(".text");
                DomElement authorEl = quote.querySelector(".author");


                if (textEl != null && authorEl != null) {
                    String text = textEl.asNormalizedText();
                    String author = authorEl.asNormalizedText();


                    System.out.println(text + " — " + author);
                }
            }


        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Before running it, replace YOUR_PROXY_USERNAME and YOUR_PROXY_PASSWORD with your actual credentials.

Once the file is ready, compile it by pointing Java to all HtmlUnit dependencies.

javac -cp "htmlunit-4.8.0/lib/*" HtmlUnitScraperWithProxy.java

javac -cp "htmlunit-4.8.0/lib/*" HtmlUnitScraperWithProxy.java

Then, run the compiled program.

java -cp ".:htmlunit-4.8.0/lib/*" HtmlUnitScraperWithProxy

java -cp ".:htmlunit-4.8.0/lib/*" HtmlUnitScraperWithProxy

The script will load a JavaScript-rendered page through the proxy, wait for it to fully render, and print the extracted quotes along with their authors.

Selenium: full browser automation

Selenium is the most powerful option that drives real browsers (Chrome, Firefox, Safari, Edge) programmatically via the WebDriver protocol, therefore interacting with the target site and mimicking human behavior, which makes it ideal for difficult scraping tasks. Selenium scrapes websites that heavily use JavaScript for dynamic content and content that is not present in the initial HTML source. Also, Selenium with Python can automate browsers to execute JavaScript and interact with pages like an actual user. 

Strengths: Handles any JavaScript, works with all major browsers, supports user interaction simulation, and has a massive community (i.e., extensive documentation, community support, and integration with CI/CD tools).

Limitations: Complex setup and high maintenance (browser binaries and drivers must be installed and maintained), high script fragility due to dependence on DOM structures (handles UI changes poorly), resource-intensive (memory/CPU), and relatively slow. 

When to use: When browser fidelity is key, for complex SPAs, target sites with login or multi-step navigation, and screenshot capture. Otherwise, tools like Playwright for Java tend to be more reliable and efficient.

Code example: Launching headless Chrome, navigating to a product page, and extracting dynamically loaded prices with Decodo proxy integration.

If you’re using Maven, add Selenium to your pom.xml file.

<dependencies>
  <dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>4.20.0</version>
  </dependency>
</dependencies>

<dependencies>
  <dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>4.20.0</version>
  </dependency>
</dependencies>

Save the following code as SeleniumScraperWithProxy.java.

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;


import java.time.Duration;
import java.util.List;


public class SeleniumScraperWithProxy {
    public static void main(String[] args) {


        // Decodo proxy details, replace with your own credentials
        String proxyHost = "gate.decodo.com";
        int proxyPort = 7000;
        String proxyUser = "YOUR_PROXY_USERNAME";
        String proxyPass = "YOUR_PROXY_PASSWORD";


        // Create proxy configuration
        Proxy proxy = new Proxy();
        proxy.setHttpProxy(proxyHost + ":" + proxyPort);
        proxy.setSslProxy(proxyHost + ":" + proxyPort);


        // Configure Chrome
        ChromeOptions options = new ChromeOptions();
        options.setProxy(proxy);


        options.addArguments("--headless=new");
        options.addArguments("--disable-gpu");
        options.addArguments("--no-sandbox");
        options.addArguments("--window-size=1920,1080");


        WebDriver driver = new ChromeDriver(options);


        try {
            driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(10));


            // Navigate to the target page
            driver.get("http://books.toscrape.com/");


            // Extract product prices
            List<WebElement> prices = driver.findElements(By.cssSelector(".price_color"));


            for (WebElement price : prices) {
                System.out.println(price.getText());
            }


        } finally {
            driver.quit();
        }
    }
}

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;


import java.time.Duration;
import java.util.List;


public class SeleniumScraperWithProxy {
    public static void main(String[] args) {


        // Decodo proxy details, replace with your own credentials
        String proxyHost = "gate.decodo.com";
        int proxyPort = 7000;
        String proxyUser = "YOUR_PROXY_USERNAME";
        String proxyPass = "YOUR_PROXY_PASSWORD";


        // Create proxy configuration
        Proxy proxy = new Proxy();
        proxy.setHttpProxy(proxyHost + ":" + proxyPort);
        proxy.setSslProxy(proxyHost + ":" + proxyPort);


        // Configure Chrome
        ChromeOptions options = new ChromeOptions();
        options.setProxy(proxy);


        options.addArguments("--headless=new");
        options.addArguments("--disable-gpu");
        options.addArguments("--no-sandbox");
        options.addArguments("--window-size=1920,1080");


        WebDriver driver = new ChromeDriver(options);


        try {
            driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(10));


            // Navigate to the target page
            driver.get("http://books.toscrape.com/");


            // Extract product prices
            List<WebElement> prices = driver.findElements(By.cssSelector(".price_color"));


            for (WebElement price : prices) {
                System.out.println(price.getText());
            }


        } finally {
            driver.quit();
        }
    }
}

Compile and run:

mvn compile
mvn exec:java -Dexec.mainClass="SeleniumScraperWithProxy"

mvn compile
mvn exec:java -Dexec.mainClass="SeleniumScraperWithProxy"

Selenium can set the proxy host and port through ChromeOptions, but Chrome does not reliably accept username and password through a simple --proxy-auth argument. That means this basic example works best for IP-whitelisted proxies.

For username and password authentication, use one of these options:

• Whitelist your machine’s IP in the proxy dashboard
• Use a browser extension to handle proxy authentication
• Use a local proxy forwarder that injects credentials

Avoid this, because it is not a valid/reliable Chrome option:

options.addArguments("--proxy-auth=username:password");

options.addArguments("--proxy-auth=username:password");

Playwright for Java: the modern alternative

Playwright for Java is a newer, faster alternative to Selenium. Playwright web scraping provides a headless browser automation tool for scraping modern, dynamic websites. It’s an open-source library developed and maintained by Microsoft that automates interactions across multiple browsers (Chromium, Firefox, and WebKit) with a single API. It enables end-to-end testing for modern web applications. Playwright is primarily written in TypeScript, so the Java version is basically a wrapper (but fully mature and supporting nearly all of Playwright's features).

Playwright doesn’t perform actions until elements are actionable (visible, stable, enabled), drastically reducing flaky tests common in other tools like Selenium. Instead of XPath or CSS selectors, which can be fragile in this context, it uses locators based on user-facing attributes. It supports BrowserContexts (isolated, lightweight browser profiles that allow parallel test execution without interference), a Test Generator (records browser actions and generates code), and a Trace Viewer (allows step-by-step post-execution debugging).

Strengths: It comes with cross-browser support, built-in parallelism, and built-in auto-wait for elements. It’s typically faster than Selenium and features a modern API design, better modern web apps handling, network interception and request mocking support, and browser isolation for each test to act like an incognito window. 

Limitations: Playwright still requires browser binaries, doesn’t support legacy browsers, and lacks a built-in test runner, while writing and maintaining scripts requires coding knowledge. It’s resource-intensive, and the newer ecosystem means fewer community resources and plugins. 

When to use: Modern, high-performance web automation, multi-browser testing, infinite scroll pages, projects focused on performance, and network-level control. 

Code example: Launching Playwright and scraping an infinite scroll page with Decodo proxy integration.

For this example, add Playwright to your pom.xml file.

<dependency>
  <groupId>com.microsoft.playwright</groupId>
  <artifactId>playwright</artifactId>
  <version>1.44.0</version>
</dependency>

<dependency>
  <groupId>com.microsoft.playwright</groupId>
  <artifactId>playwright</artifactId>
  <version>1.44.0</version>
</dependency>

Then, install the Playwright browser binaries.

mvn exec:java -e -Dexec.mainClass=com.microsoft.playwright.CLI -Dexec.args="install chromium"

mvn exec:java -e -Dexec.mainClass=com.microsoft.playwright.CLI -Dexec.args="install chromium"

Save the following code as PlaywrightUnsplashScraper.java.

import com.microsoft.playwright.*;


import java.util.List;


public class PlaywrightUnsplashScraper {
    public static void main(String[] args) {


        // Decodo proxy credentials, replace with your own
        String proxyHost = "gate.decodo.com";
        int proxyPort = 7000;
        String proxyUser = "YOUR_PROXY_USERNAME";
        String proxyPass = "YOUR_PROXY_PASSWORD";


        try (Playwright playwright = Playwright.create()) {


            Browser browser = playwright.chromium().launch(
                new BrowserType.LaunchOptions()
                    .setHeadless(true)
                    .setProxy(new Proxy()
                        .setServer("http://" + proxyHost + ":" + proxyPort)
                        .setUsername(proxyUser)
                        .setPassword(proxyPass)
                    )
            );


            BrowserContext context = browser.newContext(
                new Browser.NewContextOptions()
                    .setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) " +
                                  "AppleWebKit/537.36 (KHTML, like Gecko) " +
                                  "Chrome/124.0.0.0 Safari/537.36")
            );


            Page page = context.newPage();


            // Navigate to an infinite scroll page
            page.navigate("https://unsplash.com/s/photos/technology");
            page.waitForLoadState(LoadState.NETWORKIDLE);


            // Accept cookies if the banner appears
            try {
                page.locator("button:has-text('Accept')").click(
                    new Locator.ClickOptions().setTimeout(3000)
                );
            } catch (Exception ignored) {
            }


            // Scroll until no new images are loaded, or until the loop limit is reached
            int previousCount = 0;


            for (int i = 0; i < 10; i++) {
                page.mouse().wheel(0, 3000);
                page.waitForTimeout(2000);


                int currentCount = page.locator("figure img").count();


                if (currentCount == previousCount) {
                    break;
                }


                previousCount = currentCount;
            }


            // Extract rendered image data
            List<ElementHandle> images = page.querySelectorAll("figure img");


            for (ElementHandle img : images) {
                String src = img.getAttribute("src");
                String alt = img.getAttribute("alt");


                if (src != null && alt != null && !alt.isBlank()) {
                    System.out.println("Image: " + alt);
                    System.out.println("URL: " + src);
                    System.out.println("-----");
                }
            }


            browser.close();
        }
    }
}

import com.microsoft.playwright.*;


import java.util.List;


public class PlaywrightUnsplashScraper {
    public static void main(String[] args) {


        // Decodo proxy credentials, replace with your own
        String proxyHost = "gate.decodo.com";
        int proxyPort = 7000;
        String proxyUser = "YOUR_PROXY_USERNAME";
        String proxyPass = "YOUR_PROXY_PASSWORD";


        try (Playwright playwright = Playwright.create()) {


            Browser browser = playwright.chromium().launch(
                new BrowserType.LaunchOptions()
                    .setHeadless(true)
                    .setProxy(new Proxy()
                        .setServer("http://" + proxyHost + ":" + proxyPort)
                        .setUsername(proxyUser)
                        .setPassword(proxyPass)
                    )
            );


            BrowserContext context = browser.newContext(
                new Browser.NewContextOptions()
                    .setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) " +
                                  "AppleWebKit/537.36 (KHTML, like Gecko) " +
                                  "Chrome/124.0.0.0 Safari/537.36")
            );


            Page page = context.newPage();


            // Navigate to an infinite scroll page
            page.navigate("https://unsplash.com/s/photos/technology");
            page.waitForLoadState(LoadState.NETWORKIDLE);


            // Accept cookies if the banner appears
            try {
                page.locator("button:has-text('Accept')").click(
                    new Locator.ClickOptions().setTimeout(3000)
                );
            } catch (Exception ignored) {
            }


            // Scroll until no new images are loaded, or until the loop limit is reached
            int previousCount = 0;


            for (int i = 0; i < 10; i++) {
                page.mouse().wheel(0, 3000);
                page.waitForTimeout(2000);


                int currentCount = page.locator("figure img").count();


                if (currentCount == previousCount) {
                    break;
                }


                previousCount = currentCount;
            }


            // Extract rendered image data
            List<ElementHandle> images = page.querySelectorAll("figure img");


            for (ElementHandle img : images) {
                String src = img.getAttribute("src");
                String alt = img.getAttribute("alt");


                if (src != null && alt != null && !alt.isBlank()) {
                    System.out.println("Image: " + alt);
                    System.out.println("URL: " + src);
                    System.out.println("-----");
                }
            }


            browser.close();
        }
    }
}

Run the scraper from the project directory.

mvn compile
mvn exec:java -Dexec.mainClass="PlaywrightUnsplashScraper"

mvn compile
mvn exec:java -Dexec.mainClass="PlaywrightUnsplashScraper"

This script opens Chromium through the proxy, loads an Unsplash search results page, accepts the cookie banner if it appears, scrolls the page to trigger lazy loading, and extracts image alt text with image URLs from the rendered DOM.

Apache HttpClient + jsoup: the lightweight combo

Combining Apache HttpClient (for complex HTTP handling) with jsoup (for parsing, cleaning, extraction) is a hybrid approach that allows for control without the full browser overhead. HttpClient manages cookies, advanced requests, proxy settings, and headers, including User-Agent rotation. Therefore, it handles connections, retries, timeouts, proxies, and authentication. Also, joining Apache HttpClient with jsoup and residential proxies allows for robust Java web scraping by separating request handling from parsing while avoiding IP bans. 

Strengths: Advanced networking and separation of concerns. The HTTP requests control allows high-volume static scraping, easy proxy integration, and low resource consumption.

Limitations: It comes with the same constraints as jsoup alone. Combining Apache HttpClient with jsoup is a great solution for scraping static websites, but not modern, dynamic sites. Limitations include the inability to render JavaScript or mimic user behavior, and vulnerability to anti-bot mechanisms.

When to use: For lightweight scraping of plain HTML, as well as for large-scale static scraping, API response parsing, and websites requiring complex login, cookie management, massive parallelism, connection pooling, or custom HTTP headers.

For efficient web scraping, anonymity, and bypassing geo-restrictions, it’s best to use high-quality residential proxies. Decodo boasts a pool of 115M+ ethically-sourced residential IPs across 195+ locations worldwide, and the best response time in the market. 

To set up Decodo residential proxies with rotating IPs:

1. Register or log in to the Decodo dashboard.
2. Navigate to find residential proxies, choose a subscription, or start a 3-day free trial.
3. Go to Proxy setup.
4. Select a location or choose Random.
5. Set the rotating session type and choose a protocol (HTTP(S) or SOCKS5).
6. Choose the authentication type.
7. Download the generated endpoint and credentials or copy them into your scraper, browser, or software.

Code example: Configuring HttpClient with a Decodo residential proxy and proxy authentication, fetching HTML, and parsing with jsoup.

Add the required dependencies to your pom.xml.

<dependencies>
  <dependency>
    <groupId>org.apache.httpcomponents.client5</groupId>
    <artifactId>httpclient5</artifactId>
    <version>5.3.1</version>
  </dependency>

  <dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.17.2</version>
  </dependency>
</dependencies>

<dependencies>
  <dependency>
    <groupId>org.apache.httpcomponents.client5</groupId>
    <artifactId>httpclient5</artifactId>
    <version>5.3.1</version>
  </dependency>

  <dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.17.2</version>
  </dependency>
</dependencies>

Save the following code as HttpClientJsoupWithProxy.java.

import org.apache.hc.client5.http.auth.AuthScope;
import org.apache.hc.client5.http.auth.UsernamePasswordCredentials;
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.impl.auth.BasicCredentialsProvider;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.routing.DefaultProxyRoutePlanner;
import org.apache.hc.core5.http.HttpHost;
import org.apache.hc.core5.http.io.entity.EntityUtils;


import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;


public class HttpClientJsoupWithProxy {
    public static void main(String[] args) {


        // Decodo proxy credentials, replace with your own
        String proxyHost = "gate.decodo.com";
        int proxyPort = 7000;
        String proxyUser = "YOUR_PROXY_USERNAME";
        String proxyPass = "YOUR_PROXY_PASSWORD";


        // Target URL
        String url = "https://news.ycombinator.com/";


        try {
            // Configure proxy
            HttpHost proxy = new HttpHost(proxyHost, proxyPort);


            // Route all requests through the proxy
            DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxy);


            // Add proxy authentication
            BasicCredentialsProvider credsProvider = new BasicCredentialsProvider();
            credsProvider.setCredentials(
                new AuthScope(proxyHost, proxyPort),
                new UsernamePasswordCredentials(proxyUser, proxyPass.toCharArray())
            );


            // Build HttpClient with proxy and authentication
            try (CloseableHttpClient client = HttpClients.custom()
                    .setRoutePlanner(routePlanner)
                    .setDefaultCredentialsProvider(credsProvider)
                    .build()) {


                HttpGet request = new HttpGet(url);
                request.setHeader("User-Agent", "Mozilla/5.0");


                try (CloseableHttpResponse response = client.execute(request)) {


                    String html = EntityUtils.toString(response.getEntity());


                    // Parse the HTML response with jsoup
                    Document doc = Jsoup.parse(html, url);


                    // Extract Hacker News story titles
                    Elements titles = doc.select("span.titleline > a");


                    for (Element el : titles) {
                        String title = el.text();
                        String link = el.absUrl("href");


                        System.out.println(title);
                        System.out.println(link);
                        System.out.println("-----");
                    }
                }
            }


        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

import org.apache.hc.client5.http.auth.AuthScope;
import org.apache.hc.client5.http.auth.UsernamePasswordCredentials;
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.impl.auth.BasicCredentialsProvider;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.routing.DefaultProxyRoutePlanner;
import org.apache.hc.core5.http.HttpHost;
import org.apache.hc.core5.http.io.entity.EntityUtils;


import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;


public class HttpClientJsoupWithProxy {
    public static void main(String[] args) {


        // Decodo proxy credentials, replace with your own
        String proxyHost = "gate.decodo.com";
        int proxyPort = 7000;
        String proxyUser = "YOUR_PROXY_USERNAME";
        String proxyPass = "YOUR_PROXY_PASSWORD";


        // Target URL
        String url = "https://news.ycombinator.com/";


        try {
            // Configure proxy
            HttpHost proxy = new HttpHost(proxyHost, proxyPort);


            // Route all requests through the proxy
            DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxy);


            // Add proxy authentication
            BasicCredentialsProvider credsProvider = new BasicCredentialsProvider();
            credsProvider.setCredentials(
                new AuthScope(proxyHost, proxyPort),
                new UsernamePasswordCredentials(proxyUser, proxyPass.toCharArray())
            );


            // Build HttpClient with proxy and authentication
            try (CloseableHttpClient client = HttpClients.custom()
                    .setRoutePlanner(routePlanner)
                    .setDefaultCredentialsProvider(credsProvider)
                    .build()) {


                HttpGet request = new HttpGet(url);
                request.setHeader("User-Agent", "Mozilla/5.0");


                try (CloseableHttpResponse response = client.execute(request)) {


                    String html = EntityUtils.toString(response.getEntity());


                    // Parse the HTML response with jsoup
                    Document doc = Jsoup.parse(html, url);


                    // Extract Hacker News story titles
                    Elements titles = doc.select("span.titleline > a");


                    for (Element el : titles) {
                        String title = el.text();
                        String link = el.absUrl("href");


                        System.out.println(title);
                        System.out.println(link);
                        System.out.println("-----");
                    }
                }
            }


        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}