AI Firms Are Destroying Rare Books for Training Data: Scraping Is the Alternative

TL;DR

• As AI labs push against the limits of readily available human-written text, some have turned to buying and destructively scanning physical books for training data.
• Book scraping can collect metadata, reviews, discussions, excerpts, and public-domain text that already exists online without destroying physical copies.
• Scraping can't recover material that was never digitized, so it only covers part of the same underlying need for human-written data.
• Start with the simplest approved source available, whether that's an API, feed, bulk dataset, or public webpage, before adding heavier scraping infrastructure.

Why AI companies are running out of text

Training large language models (LLMs) takes an enormous amount of text, and most of the easiest sources, from Wikipedia and Common Crawl to books and forums, have already been used heavily.

AI can generate more training data, but that creates its own problem: models trained repeatedly on model-generated text can suffer model collapse, where the quality and diversity of the underlying data gradually degrade.

That helps explain why old books have suddenly become so valuable as AI training data. 404 Media recently reported on a shipment of rare books traced to an Amazon AI-training facility, while The Guardian covered concerns from booksellers after Anthropic was found to have bought physical books, cut off their spines, scanned the pages, and discarded the originals.

Where scraping fits as a non-destructive alternative

The rare books in those reports are valuable precisely because much of their text was never digitized. Web scraping can't recreate a book that only exists on paper, so it isn't a replacement for that kind of acquisition.

But there's also a lot of book-related material that has already made it online. Reviews, summaries, publication details, reader discussions, and publicly available excerpts can all be collected without touching the physical book. For older works, sources like Project Gutenberg also make thousands of public-domain books available digitally.

That makes scraping useful for a different, but still meaningful, part of the same problem. If the text or data you need already exists online, destroying a physical copy to get at it makes a lot less sense.

What book data you can realistically scrape

For the rest of this guide, we'll scrape Goodreads as our example and focus on a handful of fields from a public book page: the title, author, average rating, ratings count, and publicly visible reviews.

That gives us enough variety to see how book data scraping works in practice, from grabbing simple metadata to dealing with content that becomes trickier once reviews and interactions enter the picture.

Tools and setup you'll need

We'll keep the setup fairly light. For the first part of the scraper, we only need Python, Requests, and Beautiful Soup, which covers the basics of web scraping.

Later, when we get to reviews and other interactive content, we'll bring in Playwright.

First, create a virtual environment and install everything we'll need:

python3 -m venv venv
source venv/bin/activate                  # Windows: venv\Scripts\activate
python3 -m pip install requests beautifulsoup4 playwright

The virtual environment keeps the packages for this project separate from the rest of your Python setup. Once everything is installed, we can make our first request to Goodreads and see what comes back.

Writing the scraper: Extracting book fields

Let's start with a single Goodreads page and see what we can get without opening a browser. We'll use The Hobbit as our example:

Create a Python file, name it any way you want, and write this code:

import json
import requests
from bs4 import BeautifulSoup
url = "https://www.goodreads.com/book/show/5907.The_Hobbit__or_There_and_Back_Again"
response = requests.get(url, timeout=30)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
print(response.status_code)
print(soup.title.get_text(strip=True))

When you run the script with python your_script.py in the terminal, it returns a successful response and confirms that we've loaded the right page:

200
The Hobbit, or There and Back Again by J.R.R. Tolkien | Goodreads

Rather than immediately relying on CSS selectors, we can first check the structured data Goodreads includes in the page. Look for the application/ld+json block and parse it:

import json
import requests
from bs4 import BeautifulSoup
url = "https://www.goodreads.com/book/show/5907.The_Hobbit__or_There_and_Back_Again"
response = requests.get(url, timeout=30)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
schema = soup.find("script", type="application/ld+json")
book = json.loads(schema.string)
rating = book.get("aggregateRating", {})
authors = ", ".join(
   author["name"] for author in book.get("author", [])
)
book_data = {
   "title": book.get("name"),
   "author": authors,
   "average_rating": rating.get("ratingValue"),
   "ratings_count": rating.get("ratingCount"),
   "reviews_count": rating.get("reviewCount"),
}
print(book_data)

You should now see the title, author, average rating, and the current ratings and review counts for The Hobbit. At the time of testing, that returned:

{
   'title': 'The Hobbit, or There and Back Again',
   'author': 'J.R.R. Tolkien, Douglas A. Anderson, Michael Hague, Jemima Catlin',
   'average_rating': 4.3,
   'ratings_count': 4625302,
   'reviews_count': 94770
}

The counts will change over time, but the structure should remain broadly the same. 

Handling reviews and paginated or interactive content

The structured data got us the total review count, but not the individual reviews themselves. Those sit elsewhere in the page HTML, inside separate review cards.

We can pull the reviewer name, star rating, and review text from those cards with Beautiful Soup:

import requests
from bs4 import BeautifulSoup


url = "https://www.goodreads.com/book/show/5907.The_Hobbit__or_There_and_Back_Again"


response = requests.get(url, timeout=30)
response.raise_for_status()


soup = BeautifulSoup(response.text, "html.parser")
reviews = soup.select("article.ReviewCard")


print("Reviews found:", len(reviews))


for review in reviews[:3]:
   reviewer = review.get("aria-label", "").replace("Review by ", "")
   rating = review.select_one("span.RatingStars[aria-label]")
   text = review.select_one(".ReviewText .Formatted")


   print("\nReviewer:", reviewer or "Unknown")
   print("Rating:", rating.get("aria-label") if rating else "No rating")
   print(
       "Review:",
       text.get_text(" ", strip=True)[:500] if text else "No review text"
   )

That should print the number of review cards available in the initial page HTML, followed by the reviewer, rating, and first 500 characters of the first 3 reviews.

The catch is that the main book page only gives us the first set of reviews. 

Clicking "More reviews and ratings" opens a dedicated reviews page, where Goodreads loads additional results through a "Show more reviews" button rather than normal numbered pagination.

To load more than the first set, we can use Playwright to open Goodreads' dedicated reviews page and interact with the "Show more reviews" button. 

We'll use a persistent Chrome profile so the browser can retain its session between runs. Goodreads may ask you to sign in before loading additional reviews. If it does, sign in once in the Chrome window.

from pathlib import Path
from playwright.sync_api import sync_playwright
profile = str(Path.home() / ".goodreads-test")
url = "https://www.goodreads.com/book/show/5907/reviews"
target = 120
all_reviews = []
with sync_playwright() as p:
   context = p.chromium.launch_persistent_context(
       profile,
       channel="chrome",
       headless=False
   )
   page = context.pages[0] if context.pages else context.new_page()
   page.goto(url, wait_until="domcontentloaded", timeout=60000)


   page.wait_for_selector("article.ReviewCard")
   batch_number = 1
   while len(all_reviews) < target:
       reviews = page.locator("article.ReviewCard")
       batch = []


       for i in range(reviews.count()):
           review = reviews.nth(i)


           reviewer = review.get_attribute("aria-label") or "Unknown"
           rating = review.locator("span.RatingStars[aria-label]")
           text = review.locator(".ReviewText .Formatted")


           batch.append({
               "reviewer": reviewer.replace("Review by ", ""),
               "rating": (
                   rating.get_attribute("aria-label")
                   if rating.count()
                   else "No rating"
               ),
               "review": (
                   text.inner_text().strip()
                   if text.count()
                   else "No review text"
               ),
           })


       all_reviews.extend(batch)


       print(
           f"Batch {batch_number}: {len(batch)} reviews "
           f"| Total collected: {len(all_reviews)}"
       )


       if len(all_reviews) >= target:
           break


       previous_text = reviews.first.inner_text()


       button = page.locator(
           'button:has(span[data-testid="loadMore"])'
       )
       button.click()


       page.wait_for_function(
           """previous => {
               const first = document.querySelector("article.ReviewCard");
               return first && first.innerText !== previous;
           }""",
           arg=previous_text,
           timeout=30000
       )


       batch_number += 1


   all_reviews = all_reviews[:target]


   print("\nTotal reviews collected:", len(all_reviews))


   context.close()

With a target of 120, you should see the total increase as each new batch is collected:

Goodreads currently keeps 30 review cards on the page at a time, replacing the current set when another batch loads. That's why the script captures each batch before clicking "Show more reviews" again.

Storing and scaling to many books

So far, the reviews we've collected only exist while the script is running. To keep them, we can write all_reviews to a CSV file before closing the browser.

Add this just before context.close() in the previous script:

import csv
with open(
   "goodreads_reviews.csv",
   "w",
   newline="",
   encoding="utf-8"
) as file:
   writer = csv.DictWriter(
       file,
       fieldnames=["reviewer", "rating", "review"]
   )
   writer.writeheader()
   writer.writerows(all_reviews)
print("Saved reviews to goodreads_reviews.csv")

Once the script finishes, you should have a goodreads_reviews.csv file in the same folder, with one row for each review we collected.

The same idea works when you move beyond a single title. Instead of hardcoding one Goodreads URL, put the books you want into a list and run the extraction logic over each one:

import csv
import json
import time
import requests
from bs4 import BeautifulSoup


urls = [
   "https://www.goodreads.com/book/show/5907.The_Hobbit__or_There_and_Back_Again",
   "https://www.goodreads.com/book/show/61439040-1984",
   "https://www.goodreads.com/book/show/44767458-dune",
]


books = []


for url in urls:
   response = requests.get(url, timeout=30)
   response.raise_for_status()


   soup = BeautifulSoup(response.text, "html.parser")
   schema = soup.find("script", type="application/ld+json")
   book = json.loads(schema.string)


   rating = book.get("aggregateRating", {})
   authors = ", ".join(
       author["name"] for author in book.get("author", [])
   )


   books.append({
       "title": book.get("name"),
       "author": authors,
       "average_rating": rating.get("ratingValue"),
       "ratings_count": rating.get("ratingCount"),
       "reviews_count": rating.get("reviewCount"),
       "url": url,
   })


   time.sleep(2)


with open("goodreads_books.csv", "w", newline="", encoding="utf-8") as file:
   writer = csv.DictWriter(file, fieldnames=books[0].keys())
   writer.writeheader()
   writer.writerows(books)


print(f"Saved {len(books)} books to goodreads_books.csv")

That should give you a CSV with one row per book. The short delay between requests also keeps us from hitting Goodreads repeatedly as fast as the loop can run.

For a few books, CSV is perfectly fine. Once you're collecting hundreds or thousands of pages, JSON or a database usually makes more sense, and the bigger problem becomes reliably fetching all of those pages in the first place.

Staying unblocked and respecting scope at real volume

The 2-second pause we added works for a handful of pages, but it won't solve every problem once the scraper gets bigger. At higher volumes, you can start running into slower responses, 429 rate limits, 403 blocks, or pages that only load properly when JavaScript is rendered.

That's the point where it usually makes sense to stop adding more retry logic, browser handling, and proxy rotation to your own script. 

Decodo's Web Scraping API handles retries, proxy rotation, JavaScript rendering, and anti-bot measures behind the same request layer, so you can keep the extraction logic focused on the data you actually want.

The other side of scaling is knowing when to slow down. More requests don't automatically mean better data, especially if you're repeatedly hitting the same pages or collecting fields you don't need. 

Keep the scope narrow, pace requests where appropriate, and only add heavier scraping infrastructure when the target actually requires it.

Is it legal to scrape book data?

There isn't a single yes-or-no answer. Web scraping can be legal, particularly when the data is publicly accessible, but the site's terms, the type of data you collect, where you're operating, and what you plan to do with it all matter.

Before scraping at volume, check the target's terms and robots.txt, avoid collecting personal data you don't need, and respect reasonable request rates. If you're building a commercial dataset or need large amounts of copyrighted text, a licensed or explicitly permitted source is usually the safer route.

Final thoughts

Book scraping won't replace the need to digitize material that only exists in print. 

But for the huge amount of book data that is already online, it gives you a much simpler way to collect useful text, metadata, ratings, and reviews without touching the physical copy.

The practical rule is pretty simple: start with the lightest method that works, keep the scope narrow, and only add more infrastructure when the target actually demands it.