Tavily Alternatives: 7 Agentic Search APIs Compared

TL;DR: the best Tavily alternatives at a glance

• Pick by use case. Use Decodo's Web Data API for search plus full pages with one API key. Use Exa for semantic search, Perplexity for cited answers, Serper for low-cost search results, and Linkup for EU data residency.
• Search reaches only indexed pages. On 20 bot-protected sites in a benchmark, Decodo returned page content for 90%, and Tavily for 30%.
• Price per agent task. One task can trigger 5 to 15 calls, so compare credit-based and per-1K-request pricing per task.
• Keep the migration small. Put search and fetch behind one interface, then run the steps below on Decodo's free plan.

What Tavily does and why teams look for alternatives

The right alternative depends on which Tavily features you use, such as search, extraction, or crawling.

What Tavily returns and costs

Tavily's search endpoint takes a query and returns up to 20 ranked results, each with a title, a URL, a relevance score, and content snippets. With optional parameters, it also returns a short AI-generated answer and the cleaned full page.

Tavily also has extract, crawl, map, and research endpoints. Its langchain-tavily package connects it to LangChain, and its hosted Model Context Protocol (MCP) server lets AI agents call it as a tool. The older TavilySearchResults tool was the default in many LangChain templates.

The free plan gives 1K credits a month, and paid credits cost $0.005 to $0.008 each, depending on the plan. A basic search costs 1 credit and an advanced search costs 2. That keeps prototypes cheap, and costs grow quickly for high-volume production agents.

Why teams start evaluating alternatives

Teams give 4 common reasons.

• Ownership changed. Nebius announced its agreement to acquire Tavily in February 2026, and the deal closed the same month. Press reports said the price was $275M upfront, and up to $400M if performance targets are met. At the announcement, Nebius said Tavily would keep its brand and team. However, a new owner can change a product's priorities over time, so check how hard it would be to switch.
• Cost at agent scale. One user task can trigger 5 to 15 retrieval calls, a pattern called fan-out. The agent rewrites the query, searches again, and reads pages, so a price that looks small per search can become large per task.
• Extraction depth. By default, search results contain short snippets from the index. If an agent needs a whole page, a full documentation site, or a structured product record, it also needs extract or crawl calls.
• Access limits. An index can miss pages behind bot protection or logins, pages that load their content with JavaScript, and pages whose content changes by country. Tavily's country parameter ranks results from a chosen country higher, but the page content still comes from its index. The agent can then answer confidently from an outdated snippet or a price meant for another country.

Our retrieval-augmented generation (RAG) glossary entry explains how models use retrieved content, and our guide on what AI scraping is describes data collection for agents.

How to evaluate a Tavily alternative

Compare Tavily API alternatives on the 6 criteria below.

• Check which endpoints each API offers. Some APIs only search, and others can also fetch a single page, crawl a site, and map a domain's URLs. Ask whether your agent ever needs a page it didn't find through search. If it does, you also need a scraping API, which reads a page directly from its URL.
• Measure the success rate. Include pages that render with JavaScript, block bots, or change by country. Run your 5 hardest-to-scrape URLs through each free plan, and count how many return the content you need.
• Check the output format. APIs return markdown, structured JSON, or an AI-generated answer. Count the post-processing steps between the response and your vector store. Markdown also reduces token cost. On 11 pages fetched through Decodo's /scrape, markdown used 15x to 66x fewer tokens than the rendered HTML of the same pages.
• Test for outdated data and geo-targeting. Some APIs serve a cached copy by default, and some can't target a country. Compare a price in a search result with the live page, then request a page that changes by country, once from each of 2 countries. If both responses match, the API probably isn't fetching from the country you chose.
• Price a full agent task. Log 1 week of agent traffic, count searches and page fetches per completed task, and price that set of calls.
• Check framework support. Most vendors in this list offer a LangChain adapter or an MCP server, so this rarely changes the choice. Our guide to MCP servers for AI workflows lists more options. To pick the framework itself, see our comparison of AI agent frameworks.

Check whether the adapter is vendor-maintained or community-maintained, and when it was last released. Tools that exist only in the deprecated langchain-community package may not get new fixes.

The 7 best Tavily alternatives in 2026

Prices and features for these Tavily competitors come from each vendor's public pages. They change over time, so confirm them before you set a budget.

1. Decodo's Web Data API

Best for: Agents that need search plus full pages from bot-protected or country-specific sites, with one API key.

What it is: Decodo's Web Data API runs on the same network as Decodo's residential proxies and offers search, scrape, crawl, and map with one API key:

• The /search endpoint runs through the Google Search target and returns live results with URLs, titles, and snippets. It's the closest equivalent to a Tavily search call.
• The /scrape endpoint fetches one page as clean markdown or HTML, with optional JavaScript rendering and geo-targeting.
• The /crawl endpoint visits a site's pages automatically and returns their content. It can switch to JavaScript rendering for pages that need it.
• The /map endpoint discovers a site's pages in a single call, from the site and its sitemap.

The Web Data API replaces the earlier Web Scraping API and Fast Search API with one API, one key, and one subscription. Existing Web Scraping API setups continue to work, and new users start on the unified endpoints. Our introduction to the Web Data API explains each endpoint.

How it compares to Tavily: Both provide agent search, and the Web Data API can also fetch, crawl, and map with the same key, without a second vendor. Requests run on Decodo's residential proxy infrastructure and can render JavaScript. Pages that depend on JavaScript, block bots, or change by country then return full content where search-first APIs return snippets.

Benchmark against Tavily: On 50 pages, Decodo's Web Data API returned page content for 96%, and Tavily for 64%. Both returned all 10 documentation pages, so the gap came from protected sites and news.

Both ran in October 2026 with API keys from the same machine, and the same quality check was applied to every response. Decodo's column uses this article's fetch() code, and Tavily's uses its Extract and Search endpoints in the best of their basic and advanced modes. Runs at other times of day showed the same pattern. You can repeat the test on your own URLs with the shadow evaluation in the migration steps.

What it costs: You can start on the free plan, and paid plans bill per 1K requests with a money-back option. Rates are lower on larger plans, and pages without bot protection can use the lowest-cost standard proxy pool, which this article's fetch() code tries first. The pricing page lists current rates by endpoint, and the billing documentation explains how each request, including each crawled page, is billed.

When a search-only API is enough: If your agent only needs a short ranked answer and never reads the page, a search-only API does that job.

Switch to this when: Your agent needs pages that search can't reach, or you use 2 vendors, one for search and one for extraction. The Web Data API fits these use cases well:

• Crawl a documentation site into clean markdown for a RAG pipeline, in one crawl job instead of a custom crawler.
• Map an archive site to find the relevant documents first, then crawl only those pages.
• Run live web search inside an IDE or agent through the Decodo MCP server, without a separate search vendor.

Add a callback_url to the crawl request, and Decodo notifies you when a crawl finishes, so your pipeline can collect the results without polling. The same API key works with Decodo's integrations for LangChain, LlamaIndex, n8n, and Zapier, listed under Integrations in the dashboard. In the Playground, Copy agent setup copies a prompt that installs and configures Decodo's CLI and agent skill in an AI coding agent. The AI data pipelines use case shows where these endpoints fit in a production pipeline.

2. Exa

Best for: Semantic search across large document collections.

What it is: Exa is a search API built on its own embeddings-based index, so it finds results by meaning, not only by keyword match. Its endpoints include search, page contents, answers, and longer research tasks for agents.

How it compares to Tavily: Exa is stronger at open-ended discovery and less suited to general page fetching. Like Tavily, it returns LLM-ready content. Exa serves page text from its own cache by default, and its max_age_hours setting controls when it fetches the live page.

What it costs: Search costs $7 per 1K requests with the default auto type, $4 with instant, and $15 with deep-reasoning. Page contents add $1 per 1K pages for each content type, and the free plan gives $10 in credits a month.

Switch to this when: Your agent needs to find the right documents, such as "find companies building X," not read a page it already knows.

3. Firecrawl

Best for: Structured data extraction from known sites.

What it is: Firecrawl is an API for scraping, crawling, mapping, and searching, with schema-based structured output and an agent endpoint. It renders JavaScript pages and offers a LangChain package, a LlamaIndex reader, and an MCP server.

How it compares to Tavily: Where Tavily answers a question from the web, Firecrawl converts a known site into clean data. Its schema extraction and framework adapters are well developed. Its focus is extraction quality rather than access to heavily protected sites, so test your hardest URLs before you choose it.

Firecrawl also caches scrapes by default. For prices and other fast-changing pages, set maxAge to 0.

What it costs:  Firecrawl bills in credits, and a scrape, a crawled page, or a map costs 1 credit. A search costs 2 credits per 10 results, plus 1 credit for each result it scrapes. The free plan gives 1K credits a month, and the Hobby plan costs $19 a month for 5K credits.

Switch to this when: You need structured data extraction from known sites, and those sites aren't heavily protected.

4. Perplexity Search API and Agent API

Best for: Cited answers inside your product.

What it is: Perplexity's Search API returns ranked results with extracted page text, and the Agent API returns an AI-generated answer with citations. The Agent API replaced Sonar Chat Completions.

How it compares to Tavily: With the Agent API, Perplexity does more of the reasoning, so your pipeline does less work but controls less of what enters your context window. The Search API is the closer Tavily equivalent.

What it costs: The Search API costs $5 per 1K requests, or $1 per 1K in its Fast Search mode. The Agent API bills model tokens plus tool fees, such as $1 to $2.50 per 1K web searches. At high volume, that bill grows with answer length and search count, unlike a fixed credit cost per call.

Switch to this when: You want a cited answer in your product, and you accept that Perplexity handles search and answer generation.

5. Brave Search API

Best for: Search on an independent index with a privacy focus.

What it is. Brave Search API has web, news, image, video, and place endpoints, plus an LLM Context endpoint that returns page chunks for models.

How it compares to Tavily: Unlike Tavily's search, Brave's web endpoint returns standard search results, so you add cleaning and chunking yourself or use the LLM Context endpoint. Storing results long-term, for example to fine-tune a model, needs a plan that includes storage rights.

What it costs: Search costs $5 per 1K requests. Accounts get $5 in free credits a month, and Brave requires a credit card for identity checks.

Switch to this when: An independent index, a single per-request price, or privacy matters more than LLM-ready output. Its original LangChain tool was in the deprecated langchain-community package, so check whether a maintained one exists.

6. Linkup

Best for: Teams that need a European vendor or data residency.

What it is: Linkup is a French search API that focuses on fast responses, with search depths from flash to deep. It returns raw results, an answer with sources, or JSON in a schema you define.

How it compares to Tavily: Linkup does a similar job to Tavily, and it differs in jurisdiction and contract terms. Queries can run in any of its regions by default. Guaranteed processing in one region is an enterprise option.

What it costs: Standard search costs $5 per 1K requests, and deep search costs $50 to $55 per 1K. New accounts with a work email get $20 in credits, reset to $20 each month for eligible accounts.

Switch to this when: Your procurement or compliance team needs a European vendor, or data residency written into the contract.

7. Serper.dev

Best for: Low-cost Google result links for your own extraction stack.

What it is: Serper is a minimal Google results API. It returns Google search engine results pages (SERPs) as structured JSON for web, news, images, maps, shopping, and other result types.

How it compares to Tavily: Serper returns raw results, and your pipeline handles content cleaning, answer generation, and formatting for agents. Its per-query price is among the lowest in this list for search results alone, and its original LangChain wrapper was in the deprecated langchain-community package.

What it costs: Credit packs cost from $1.00 per 1K queries on the smallest pack to $0.30 on the largest. Credits expire after 6 months, and new accounts get 2.5K free queries.

Switch to this when: You already run your own extraction and cleaning stack and only need result links at a low cost. Include the engineering time for that stack in your budget.

To collect parsed Google results at scale, use the Google Search Scraper API. Our guide to AI data collection tools lists more options in this category.

Tavily alternatives compared

Prices in this article are list prices as of October 2026.

What none of these APIs solve on their own

Search results come from an index built before your query, whichever API you use. That's why the Web Data API combines search with /scrape, which reads the live page. When an agent gives a confident, wrong answer, check whether search or page fetching failed before you change vendors.

• Outdated index. For one Best Buy product page, the Google result snippet still showed a $1,099.99 price that the page no longer listed. Fetched live with Decodo's /scrape in the same test session, the page showed $764.99. Serving cached copies is a reasonable design choice, because keeping an index up to date is expensive. Exa, Firecrawl, and Tavily each let you set a maximum age for cached results. For price tracking, fetch the live page instead of trusting a cached snippet.
• Protected pages. Pages behind logins or strict bot detection, and single-page apps that render with JavaScript, can give search APIs partial content or nothing.
• Location-specific content. The same URL can return different content by country, and without geo-targeting, an agent can answer for the wrong market without any warning. Decodo's /scrape fetched the Steam store page for Stardew Valley with geo set to 4 countries. The game cost $10.49 in the US, 9,79€ in Germany, ₹ 335 in India, and ¥ 1,036 in Japan.
• Fan-out cost. When retrieval misses a page, an agent may search again. Each retry adds another billed call and more tokens, so per-call costs grow without a clear warning.

For outdated indexes, protected pages, and location-specific content, combine search with a scraping API that fetches the live page from the right country. Decodo's /scrape does this with JavaScript rendering and geo. Our guide on crawling vs. scraping explains how the 2 methods differ.

For fan-out cost, cache results in your agent with an expiry time per content type, and keep it short for prices and stock levels. A repeated question then doesn't trigger a new search.

How to scrape a website to markdown shows the fetch step in code. How to find all URLs on a domain shows how to list a site's pages first.

How to migrate off Tavily

The 5 steps below are designed so you can reverse each change.

Step 1. Abstract the dependency

Put retrieval behind one internal interface with 2 methods: search(query) for ranked results and fetch(url) for a full page. Changing the vendor then means changing a config value, not refactoring code.

Step 2. Replace the tool binding

In LangChain, TavilySearch is one entry in the list of tools that you pass to the agent. Replace it with 2 tools, web_search and fetch_page, that both call the interface.

The Tavily tool also lets the model filter by domain and by date. Keep those filters in the new tool, or answers that depended on them can get worse without any error. The full code at the end of this article maps each common Tavily option to its Web Data API equivalent.

Keep the tool descriptions short, because the model reads them on every call. The Web Data API documentation explains parameters and authentication, and the Decodo LangChain integration page describes Decodo's LangChain tools.

Treat fetched pages as untrusted input. A page can contain instructions written for the model, known as prompt injection. Don't let fetched content alone trigger tools that send messages or change data.

Step 3. Recalculate cost

Convert your credit usage into calls per completed task, because credit prices and per-1K prices don't compare directly. Then price that volume on the Web Data API pricing page.

Before you convert, check which search depth your Tavily calls use, because advanced search costs 2 credits and basic costs 1. The deprecated TavilySearchResults defaults to advanced, and the auto_parameters option can upgrade a basic search to advanced.

Step 4. Run a shadow evaluation

Send the same 100 production queries to both setups and compare which one returns the pages your agent needed. Then send the URLs that your agent fetched, and count how many returned a full page, a truncated snippet, or a block page.

On the Web Data API, you can start the shadow evaluation on the free plan. Measure completeness rather than latency, because a CAPTCHA page passed to the model as content can change the agent's answer.

Step 5. Keep a fallback

Keep the old provider behind the interface for 1 release cycle, so you can switch back with one config change if live results differ. Remove the old provider after live traffic confirms the shadow evaluation results.

For production patterns beyond the migration, see our guide to building a production RAG pipeline with LlamaIndex. The AI agent enablement use case shows where web access fits in an agent stack.

Full code for the migration example

You can run the 3 files below on the free Web Data API plan. They need Python 3.10 or later and a Web Data API key, which you get in the Decodo dashboard when you activate the free plan. Install the dependencies and set your key:

pip install httpx langchain-core
pip install tavily-python
export DECODO_AUTH_TOKEN="your-api-key"

Set TAVILY_API_KEY only if you keep Tavily as a fallback. The same code, plus a script that checks your setup, is in the web-data-api-tavily-migration repo. If you prefer a typed client library, see our Python SDK.

The _post() helper adds retries with backoff, a standard pattern for any HTTP API. Save the retrieval interface as retrieval.py:

"""A retrieval interface that makes the web search vendor a config value.

Set RETRIEVER=decodo or RETRIEVER=tavily. The agent code only sees search() and fetch().
"""
import os
import re
import time
from dataclasses import dataclass
from typing import Protocol

import httpx

BLOCK = re.compile(r"(?i)captcha|access denied|verify you are human|enable javascript|unusual traffic")
MIN_PAGE_CHARS = 2_000  # Below this, treat the result as a snippet, an error page, or a page that still needs rendering
LINK_TARGETS = re.compile(r"!\[[^\]]*\]\([^)]*\)|(?<=\])\([^)]*\)")  # Images and link URLs, which aren't readable text
TIME_RANGES = {"day": "qdr:d", "week": "qdr:w", "month": "qdr:m", "year": "qdr:y"}
RETRY_STATUS = {429, 500, 502, 503, 504}


@dataclass
class Doc:
    url: str
    title: str
    text: str
    kind: str  # "snippet" for a search result, "page" for fetched content


def page_quality(text: str) -> str:
    """Label fetched text as complete, partial, or blocked."""
    if BLOCK.search(text[:5_000]):
        return "blocked"
    # Count readable text only, because a page shell full of menu links can be long and still have no content
    return "complete" if len(LINK_TARGETS.sub("", text)) >= MIN_PAGE_CHARS else "partial"


class Retriever(Protocol):
    def search(self, query: str, max_results: int = 5, domains: list[str] | None = None,
               time_range: str | None = None) -> list[Doc]: ...
    def fetch(self, url: str) -> Doc: ...


class DecodoRetriever:
    """Decodo Web Data API: parsed Google results for search, /scrape for the page itself."""

    def __init__(self, api_key: str | None = None, geo: str = "United States", timeout: float = 150.0):
        key = api_key or os.environ["DECODO_AUTH_TOKEN"]
        self.geo = geo
        self.http = httpx.Client(
            base_url="https://data.decodo.com/v1",
            headers={"Authorization": f"Bearer {key}", "Accept": "application/json"},
            timeout=timeout,
        )

    def _post(self, path: str, body: dict, attempts: int = 3) -> dict:
        for attempt in range(1, attempts + 1):
            try:
                resp = self.http.post(path, json=body)
                resp.raise_for_status()
                data = resp.json()
                if data.get("status") in ("failed", "faulted"):
                    raise RuntimeError(f"{path} failed: {data.get('message')}")
                return data
            except (httpx.TransportError, httpx.HTTPStatusError, RuntimeError) as exc:
                if isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code not in RETRY_STATUS:
                    raise  # Don't retry a bad request or a bad key
                if attempt == attempts:
                    raise
                time.sleep(2 ** attempt)  # Wait 2 s, then 4 s

    def search(self, query: str, max_results: int = 5, domains: list[str] | None = None,
               time_range: str | None = None) -> list[Doc]:
        if domains:
            query += " (" + " OR ".join(f"site:{d}" for d in domains) + ")"
        # google_nfpr disables autocorrect, so Google searches the exact query
        body = {"target": "google_search", "query": query, "parse": True, "geo": self.geo, "google_nfpr": True}
        if time_range:
            body["google_tbs"] = TIME_RANGES[time_range]
        data = self._post("/scrape", body)
        organic = data["results"][0]["content"]["results"]["results"].get("organic", [])
        return [
            Doc(url=r["url"], title=r.get("title", ""), kind="snippet",
                text=r.get("desc", "").removesuffix("Read more"))  # Remove Google's "Read more" label
            for r in organic[:max_results]
        ]

    def _scrape(self, url: str, options: dict, attempts: int = 3) -> str:
        body = {"url": url, "markdown": True, "geo": self.geo, **options}
        try:
            data = self._post("/scrape", body, attempts)
        except httpx.HTTPStatusError as exc:
            if exc.response.status_code != 400:
                raise
            body.pop("geo")  # Retry once with the default location
            data = self._post("/scrape", body, attempts)
        result = data["results"][0]
        text = result.get("content") or ""
        if not text:  # Some sites send a real page with a 403, so judge the content, not the code
            raise RuntimeError(f"{url} returned no content (HTTP {result.get('status_code')})")
        return text

    def fetch(self, url: str) -> Doc:
        # Try the lowest-cost settings first, and add JavaScript rendering only for pages that need it
        steps = [{"proxy_pool": "standard"}, {"proxy_pool": "premium"},
                 {"proxy_pool": "premium", "headless": "html"}]
        for i, options in enumerate(steps):
            last = i == len(steps) - 1
            try:
                text = self._scrape(url, options, attempts=3 if last else 1)
            except (httpx.HTTPError, RuntimeError):
                if last:
                    raise
                continue
            if last or page_quality(text) == "complete":
                return Doc(url=url, title="", text=text, kind="page")


class TavilyRetriever:
    """The existing Tavily setup, kept behind the same interface as a fallback."""

    def __init__(self, api_key: str | None = None):
        from tavily import TavilyClient

        self.client = TavilyClient(api_key=api_key or os.environ["TAVILY_API_KEY"])

    def search(self, query: str, max_results: int = 5, domains: list[str] | None = None,
               time_range: str | None = None) -> list[Doc]:
        data = self.client.search(query, max_results=max_results, include_domains=domains, time_range=time_range)
        return [
            Doc(url=r["url"], title=r.get("title", ""), text=r.get("content", ""), kind="snippet")
            for r in data.get("results", [])
        ]

    def fetch(self, url: str) -> Doc:
        data = self.client.extract(urls=[url], format="markdown")
        results = data.get("results", [])
        if not results:
            raise RuntimeError(f"{url} could not be extracted")
        return Doc(url=url, title="", text=results[0].get("raw_content") or "", kind="page")


def get_retriever() -> Retriever:
    name = os.environ.get("RETRIEVER", "decodo")
    return {"decodo": DecodoRetriever, "tavily": TavilyRetriever}[name]()

The tools in agent_tools.py keep the common Tavily options:

Save the LangChain tools as agent_tools.py:

"""LangChain tools that replace the Tavily search tool in an agent's tool list."""
from typing import Literal

from langchain_core.tools import tool

from retrieval import get_retriever, page_quality

retriever = get_retriever()
MAX_PAGE_CHARS = 12_000  # Limit each fetched page so it doesn't fill the context window


@tool(parse_docstring=True)
def web_search(query: str, domains: list[str] | None = None,
               time_range: Literal["day", "week", "month", "year"] | None = None) -> str:
    """Search the live web. Returns up to 5 results with title, URL, and snippet.

    Args:
        query: The search query.
        domains: Only return results from these domains, such as ["github.com"].
        time_range: Only return results from the last day, week, month, or year.
    """
    try:
        docs = retriever.search(query, max_results=5, domains=domains, time_range=time_range)
    except Exception as exc:  # Return the error to the agent instead of ending the run
        return f"Search failed for {query!r}: {exc}"
    return "\n\n".join(f"{d.title}\n{d.url}\n{d.text}" for d in docs) or "No results."


@tool
def fetch_page(url: str) -> str:
    """Fetch one URL and return its main content as markdown."""
    try:
        text = retriever.fetch(url).text
    except Exception as exc:  # Return the error to the agent instead of ending the run
        return f"Fetch failed for {url}: {exc}"
    if page_quality(text) == "blocked":  # Don't pass a CAPTCHA or block page to the model as content
        return f"Fetch failed for {url}: the site returned a block page instead of content."
    start = text.find("\n# ")  # Skip site navigation when the page has a title heading
    return text[max(start, 0):][:MAX_PAGE_CHARS].strip()


if __name__ == "__main__":
    results = web_search.invoke({"query": "fastapi dependency injection tutorial"})
    print(results.split("\n\n")[0])  # First result
    print("---")
    print(fetch_page.invoke({"url": "https://fastapi.tiangolo.com/tutorial/dependencies/"})[:250])

Pass [web_search, fetch_page] wherever your agent previously received [TavilySearch()]. Run python agent_tools.py to try both tools. The output starts like this:

Global Dependencies
https://fastapi.tiangolo.com/tutorial/dependencies/global-dependencies/
Similar to the way you can add dependencies to the path operation decorators, you can add them to the FastAPI application. In that case, they will be applied to ...
---
# Dependencies[¶](#dependencies "Permanent link")

**FastAPI** has a very powerful but intuitive **Dependency Injection** system.

It is designed to be very simple to use, and to make it very easy for any developer to integrate other components with

Save the shadow evaluation as shadow_eval.py:

"""Send the same production URLs to two retrievers and check how complete each page is.

Usage: python shadow_eval.py urls.txt
Each line of urls.txt is one URL your agent needed in production.
"""
import os
import sys

from retrieval import DecodoRetriever, TavilyRetriever, page_quality


def check_page(retriever, url: str) -> str:
    try:
        text = retriever.fetch(url).text
    except Exception as exc:
        return f"error ({type(exc).__name__})"
    return f"{page_quality(text)} ({len(text)} chars)"


def main(path: str) -> None:
    with open(path) as f:
        urls = [line.strip() for line in f if line.strip()]
    retrievers = {"decodo": DecodoRetriever()}
    if os.environ.get("TAVILY_API_KEY"):
        retrievers["tavily"] = TavilyRetriever()
    totals = {name: 0 for name in retrievers}
    for url in urls:
        row = {name: check_page(retriever, url) for name, retriever in retrievers.items()}
        for name, label in row.items():
            totals[name] += label.startswith("complete")
        print(url, row)
    for name, n in totals.items():
        print(f"{name}: {n} of {len(urls)} complete")


if __name__ == "__main__":
    main(sys.argv[1])

Put URLs in urls.txt, one per line, and run python shadow_eval.py urls.txt. The output looks like this:

https://fastapi.tiangolo.com/tutorial/ {'decodo': 'complete (19464 chars)'}
https://www.glassdoor.com/Reviews/Google-Reviews-E9079.htm {'decodo': 'complete (24584 chars)'}
decodo: 2 of 2 complete

Use 100 URLs from your own agent logs before you decide, because completeness depends on which sites your agent visits.

Final thoughts

Tavily is a capable default, and the right alternative depends on your use case. Use a search-only API when your agent answers questions from snippets, and a semantic search tool when it needs to find documents by meaning. When the agent needs the page and not only the snippet, Decodo's Web Data API provides search and page fetch with one API key.

Start by running your 5 hardest-to-scrape production URLs on the Web Data API free plan and counting how many return complete content. Whichever vendor you pick, price the full agent task and keep the retrieval layer replaceable.