Back to blog

How to Build an LLM Visibility Tracker in Python

Share article:

An LLM visibility tracker sends the same prompts to AI answer engines on a schedule and logs whether each answer mentions your brand, cites your site, and names competitors. Buyers ask ChatGPT and Google AI Mode for recommendations, and answers vary from run to run, so one check says little. Build it in Python on Decodo's AI templates, code included.

Glowing neon document icon with an eye symbol on a dark, wavy background.

TL;DR

  • Measure LLM visibility as mention rate, citation rate, and share of voice across many prompts, never from a single reply.
  • Run every prompt several times per engine, because AI answers change from run to run.
  • Collect ChatGPT and Google AI Mode answers as parsed JSON through the Decodo Web Data API, and store every run in SQLite.
  • Read category prompts first, since brand and comparison prompts name your brand and inflate its mention rate.

What is an LLM visibility tracker?

An LLM visibility tracker is a measurement layer for AI answers, turning what answer engines say about a brand into numbers you can compare over time. The "LLM" is the large language model behind each engine.

It replaces the manual routine of typing prompts into each assistant and screenshotting the replies, a system that scales about as well as sticky notes on a monitor.

Two neighboring tools do different jobs. An SEO rank tracker records where a page sits in a list of links, and AI answers have no such list, only generated text. LLM monitoring and observability tools watch your own AI app, including its response time, token usage, and errors, while a visibility tracker checks what public assistants say about your brand.

How to measure LLM visibility

You measure LLM visibility as rates across many prompts and repeated runs: how often answers mention your brand, how often they cite your domain, and what share of tracked-brand mentions is yours.

Brand mention rate

Brand mention rate is the share of answers that name your brand or an alias, calculated per prompt and per engine. Tracking brand mentions in AI answers means matching more than the official name, so the alias list should cover product names, former brand names, and common misspellings. An invented brand called Brightloom might also appear as "Brightloop" (its old name), "Loom Pro" (a product), or "Brigthloom" (a typo that never got fixed).

LLM citation tracking

LLM citation tracking records which URLs an answer cites as sources and whether any of them belong to your domain. A mention is the brand name in the text, while a citation is a link to a page, and an answer can mention you while citing a competitor's review of you.

Matching is where this goes sideways. Compare domains, not full URLs, and strip tracking parameters first. ChatGPT appends utm_source=chatgpt.com to cited links, so an exact URL match against your own page would fail.

AI share of voice

AI share of voice is your brand's mentions divided by all mentions of the brands you track, over the same prompts and runs. Count each answer once per brand, so a chatty paragraph doesn't outvote a short one.

The figure is only as complete as the competitor list. A rival missing from it quietly inflates your share, which feels great until it doesn't, so the tracker should let users add competitors and aliases without touching code.

Position and sentiment

First-mention position can be recorded, but report it as a distribution, not a rank. SparkToro's January 2026 study of ChatGPT, Claude, and Google's AI found a less than 1-in-100 chance of getting the same brand list twice, and about 1 in 1,000 for the same list in the same order. The sampling section below covers what that means for the tracker.

Sentiment is optional. Classifying it takes another model call per answer and adds its own error rate, so the tracker in this guide leaves it as an extension point.

The table below sums up the LLM visibility metrics the tracker reports:

Metric

What it counts

How to calculate

Watch out for

Mention rate

Answers that name the brand or an alias

Runs mentioning the brand divided by all runs, per prompt and engine

Aliases, misspellings, brand names inside other words

Citation rate

Answers that cite a URL on the brand's domain

Runs citing the domain divided by all runs

ChatGPT returns citations only when the answer cites sources, and tracking parameters need stripping

AI share of voice

The brand's share of mentions among tracked brands

Brand mentions divided by mentions of all tracked brands

Only as complete as the competitor list

First-mention position

Where the brand first appears in the answer

Distribution of positions across runs

Unstable run to run, so never report it as a rank

Sentiment (optional)

How the answer describes the brand

Classifier or rules per answer

Extra model cost and its own error rate

These numbers also show whether any AI optimization (AIO) work changes anything.

Why one run per prompt isn't enough

AI answers are generated, not looked up, so the same prompt can return a different answer every time you send it. SparkToro's study tested this by having volunteers send identical prompts repeatedly and comparing the results. In its headphones example, brands appeared in 55 to 77% of 994 responses, so a brand can appear in one run and be missing from the next.

Search Engine Land's LLM optimization guide describes the fix as a polling-based model inspired by election forecasting. It recommends sampling 250 to 500 high-intent queries, run daily or weekly. No pollster phones one voter and declares a winner, and a tracker shouldn't either.

So the tracker sends each prompt several times per engine. Runs per prompt is configurable and defaults to 3 to 5; each run is stored separately, and the metrics above are calculated across all of them.

More runs mean more calls, and the math adds up fast. Take 50 prompts across two engines at five runs each, which is 500 calls per cycle. A weekly schedule comes to about 2,170 calls a month, while a daily one reaches about 15,000. Start with three runs and a weekly cadence, then raise the run count only for the prompts that matter most or swing the most between runs.

What the tracker collects from ChatGPT and Google AI Mode

The tracker gets its data from Decodo's Web Data API, which offers ready-made templates for AI answer engines. Send a prompt to the matching template, and it returns the engine's answer as parsed JSON, with the answer text and its sources, so you don't have to drive browser tabs or copy and paste replies. The two engines differ in ways that shape the code:

  • ChatGPT. The ChatGPT scraping template returns citations only with web search turned on (search: true). Without it, the answer reflects what the model already knows, which is worth tracking separately.
  • Google AI Mode. It takes a query parameter instead of a prompt and is only available in countries where Google offers AI Mode, so check your target markets first.

Both templates accept a geo parameter, so you can compare the same prompt across markets. The same pattern extends to Gemini through the gemini template, but this guide sticks to two engines.

The table below shows what each template returns:

ChatGPT

Google AI Mode

Template

chatgpt

google_ai_mode

Input

prompt (under 4,000 characters)

query

Integration

Asynchronous only, batch supported

Real-time or asynchronous, batch supported

Answer text

response_text, markdown_text

response_text

Sources

citations (URL, title, text) and links, when the answer cites sources

citations (source URLs and text) and links

Engine-specific options

search: true turns on web search

geo limited to supported countries

Other fields

llm_model, markdown_json

None listed

Source: Decodo documentation for ChatGPT and Google AI Mode, checked November 20, 2026.

Query AI engines at scale

Send prompts to ChatGPT and Google AI Mode through the Web Data API and get parsed JSON back.

Build the tracker step-by-step

By the end of these eight steps, you'll have a single Python file that asks three AI engines your questions, checks whether they name you, and keeps the receipts in SQLite.

1. Set up the project

You need Python 3.11 or newer (the config uses the standard library's tomllib), plus Requests and python-dotenv:

mkdir llm-visibility-tracker && cd llm-visibility-tracker
python -m venv venv && source venv/bin/activate
pip install requests python-dotenv

Virtual environments and .env files work the same way as in our rank tracker guide, with one difference. This tracker needs a Web Data API key from the dashboard, because legacy Web Scraping API tokens don't work with it. Save the key in .env and keep that file out of version control:

DECODO_API_KEY=YOUR_WEB_DATA_API_KEY

Then confirm the key works with a one-off script, check_key.py:

import os
import requests
from dotenv import load_dotenv


load_dotenv()
response = requests.post(
    "https://data.decodo.com/v1/task",
    headers={
        "Authorization": f"Bearer {os.environ['DECODO_API_KEY']}",
        "Accept": "application/json",
        "Content-Type": "application/json",
    },
    json={"target": "chatgpt", "prompt": "Reply with one word: ready", "parse": True},
    timeout=60,
)
print(response.status_code, response.text[:300])

Run python check_key.py. A 200 status with a task ID and "status":"pending" means the key works and the task is queued. An error status such as 401 means the key was rejected. The script submits one real task, so it costs one request.

2. Define brands and prompts

Everything the tracker watches lives in config.toml: your brand, its competitors, the engines, the markets, and the prompts, each tagged by type (brand, category, or comparison) and market. Quillvane is an invented brand on reserved .example domains, so swap in your own. Save this as config.toml:

# Sample config. Quillvane, Ledgerwren, and Tarnmere are invented brands.
# The .example domains are reserved for documentation and never resolve.


[settings]
runs_per_prompt = 3          # how many times each prompt is sent to each engine per cycle
max_workers = 5              # tasks in flight at once
poll_interval_seconds = 5    # seconds between status checks for a pending task
task_timeout_seconds = 300   # give up on a task after this long


# Market keys used by the prompts below, mapped to the country name sent as `geo`.
[markets]
us = "United States"


# One entry per engine. `params` are merged into every request for that engine.
# To track ChatGPT without web search as a separate series, add a second entry
# with a different name and `params = { search = false }`.
[[engines]]
name = "chatgpt"
target = "chatgpt"
params = { search = true }


[[engines]]
name = "google_ai_mode"
target = "google_ai_mode"


[brand]
name = "Quillvane"
aliases = ["Quillvain", "Vane Pro", "Quilvane"]
domains = ["quillvane.example"]


[[competitors]]
name = "Ledgerwren"
aliases = []
domains = ["ledgerwren.example"]


[[competitors]]
name = "Tarnmere"
aliases = ["Tarn Mere"]
domains = ["tarnmere.example"]


# Prompt types: brand, category, comparison.
[[prompts]]
text = "Is Quillvane any good for freelancers?"
type = "brand"
market = "us"


[[prompts]]
text = "What is the best invoicing software for freelancers?"
type = "category"
market = "us"


[[prompts]]
text = "Which invoicing tools handle recurring invoices and late payment reminders well?"
type = "category"
market = "us"


[[prompts]]
text = "Quillvane vs Ledgerwren: which is better for a solo designer?"
type = "comparison"
market = "us"

ChatGPT gets search = true, so its answers come with web citations. To track it without search as a separate series, add a second engine entry with a different name and search = false.

Now create llm_tracker.py and start it with the imports for the whole project:

"""LLM visibility tracker: collect AI answers through the Decodo Web Data API and measure brand visibility."""
import argparse
import csv
import json
import os
import re
import sqlite3
import sys
import threading
import time
import tomllib
from collections import Counter
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from datetime import datetime, timezone
from functools import lru_cache
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit


import requests
from dotenv import load_dotenv

Then add the config loader. It fills in default settings, stops on an undefined market, and puts your brand first in the brand list:

def load_config(path):
    with open(path, "rb") as fh:
        cfg = tomllib.load(fh)
    cfg["settings"] = {
        "runs_per_prompt": 3,
        "max_workers": 5,
        "poll_interval_seconds": 5,
        "task_timeout_seconds": 300,
        **cfg.get("settings", {}),
    }
    for prompt in cfg["prompts"]:
        if prompt["market"] not in cfg["markets"]:
            sys.exit(f"Prompt market '{prompt['market']}' is not defined under [markets]")
    # Own brand first, then competitors. Domains are lowercased, without a leading www.
    brands = [{**cfg["brand"], "is_own": True}] + [{**c, "is_own": False} for c in cfg.get("competitors", [])]
    for brand in brands:
        brand["domains"] = [d.lower().removeprefix("www.") for d in brand.get("domains", [])]
    cfg["brands"] = brands
    return cfg

Check that it loads:

python -c "import llm_tracker as t; print(t.load_config('config.toml')['brands'][0]['name'])"

The command should print "Quillvane".

3. Submit tasks

Every code block from here on goes at the bottom of llm_tracker.py, below the previous step's code. Each task is one prompt, one engine, and one run, submitted asynchronously with parse on, so the answer comes back as structured fields. Start with the constants and a guard that stops when the key is missing or still the placeholder:

_local = threading.local()  # one HTTP session per worker thread


def session():
    if not hasattr(_local, "session"):
        _local.session = requests.Session()
        _local.session.headers.update(
            {
                "Authorization": f"Bearer {os.environ['DECODO_API_KEY']}",
                "Accept": "application/json",
                "Content-Type": "application/json",
            }
        )
    return _local.session

The HTTP layer gives each worker thread its own session and retries rate limits and server errors up to three times:

def api(method, path, **kwargs):
    """Call the Web Data API, retrying rate limits and server errors."""
    for attempt in range(1, 4):
        last = attempt == 3
        try:
            resp = session().request(method, f"{BASE_URL}{path}", timeout=60, **kwargs)
        except requests.RequestException as exc:
            if last:
                raise ApiError(f"{method} {path} failed: {exc}") from exc
            time.sleep(2**attempt)
            continue
        if resp.status_code in (429, 500, 502, 503, 504) and not last:
            time.sleep(2**attempt)
            continue
        if resp.status_code >= 400:
            raise ApiError(f"{method} {path} returned {resp.status_code}: {resp.text[:200]}")
        return resp.json()

submit() creates a task, with Google AI Mode's query quirk handled in one line. wait_for_results() polls until the task is done and raises an error if it faults or times out. The asynchronous requests docs cover the lifecycle.

def submit(target, text, geo, params):
    body = {"target": target, PROMPT_FIELD.get(target, "prompt"): text, "parse": True, "geo": geo, **params}
    return str(api("POST", "/task", json=body)["id"])




def wait_for_results(task_id, timeout, interval):
    deadline = time.monotonic() + timeout
    while True:
        status = api("GET", f"/task/{task_id}").get("status")
        if status == "done":
            return api("GET", f"/task/{task_id}/results")
        if status == "faulted":
            item = (api("GET", f"/task/{task_id}/results").get("results") or [{}])[0]
            raise ApiError(f"Task {task_id} faulted (status code {item.get('status_code')})")
        if time.monotonic() >= deadline:
            raise ApiError(f"Task {task_id} still '{status}' after {timeout:.0f}s")
        time.sleep(interval)

The Google AI Mode template also supports batch requests, but individual tasks keep each failure easy to trace.

4. Normalize responses

Each engine nests its answer differently, so the normalizer maps all three into one record, and nothing after this step touches an engine-specific field. First, a helper that strips tracking parameters from cited URLs, so one page isn't counted as several:

TRACKING_PARAMS = {"gclid", "fbclid"}




def clean_url(url):
    """Drop tracking parameters (utm_*, gclid, fbclid) and the fragment."""
    if not url.startswith(("http://", "https://")):
        return url.strip()                                                      
    parts = urlsplit(url.strip())
    query = [
        (k, v)
        for k, v in parse_qsl(parts.query, keep_blank_values=True)
        if not k.lower().startswith("utm_") and k.lower() not in TRACKING_PARAMS
    ]
    return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), ""))

Then the record and a helper that finds the answer fields. The Web Data API pages show them directly under content, while the older ChatGPT page nests them deeper, so the helper accepts both:

@dataclass
class Record:
    engine: str
    prompt: str
    prompt_type: str
    run: int
    market: str
    collected_at: str
    text: str
    cited_urls: list
    raw: dict




def content_of(payload):
    """Return the dict holding the answer fields.


    The Web Data API pages show `content` holding the fields directly. The older
    ChatGPT page nests them one level deeper under `content.results`, so accept both.
    """
    results = payload.get("results") or []
    if not results:
        raise ValueError("Response has no results")
    content = results[0].get("content") or {}
    inner = content.get("results")
    return inner if isinstance(inner, dict) else content

ChatGPT and Google AI Mode keep their text and sources in plain fields, with one catch each that showed up in real runs. With search on, ChatGPT can append sponsored blocks to response_text, followed by ad data that ran past 80,000 characters in some of my runs. That would feed competitors' ad copy into your mention counts, so from_chatgpt() cuts the text where the first ad begins. Google AI Mode returns most of its sources as google.com/goto redirects that hide the real site. The links list names the publisher behind each one, so from_google_ai_mode() keeps that name instead of the redirect. 

def from_chatgpt(content):
    urls = [c["url"] for c in content.get("citations") or [] if c.get("url")]
    text = content.get("response_text") or ""
    # With search on, sponsored blocks can trail the answer, followed by ad data that can
    # run to tens of thousands of characters. Cut the text where the first ad block starts.
    marker = text.find("About this ad")
    if marker != -1:
        start = text.rfind("Learn more", 0, marker)
        text = text[: start if start != -1 else marker].rstrip()
    return text, urls


def is_redirect(url):
    parts = urlsplit(url)
    return (parts.hostname or "").removeprefix("www.") == "google.com" and parts.path.startswith("/goto")


def from_google_ai_mode(content):
    # Most sources arrive as google.com/goto redirects, which hide the real site. The
    # "links" list names the publisher behind each one, so keep that label instead.
    links = content.get("links") or []
    publishers = {link["url"]: link.get("publisher") for link in links if link.get("url")}
    urls = [u for citation in content.get("citations") or [] for u in citation.get("urls") or [] if u]
    sources = []
    for url in urls + list(publishers):
        if not is_redirect(url):
            sources.append(url)
        elif publishers.get(url):
            sources.append(publishers[url])
    return content.get("response_text") or "", sources


EXTRACTORS = {"chatgpt": from_chatgpt, "google_ai_mode": from_google_ai_mode}

build_record() picks the extractor and rejects an answer with no text instead of storing an empty row:

def build_record(engine_name, target, prompt, run, payload):
    text, urls = EXTRACTORS[target](content_of(payload))
    if not text.strip():
        raise ValueError(f"{engine_name} returned no answer text (is the market supported?)")
    return Record(
        engine=engine_name,
        prompt=prompt["text"],
        prompt_type=prompt["type"],
        run=run,
        market=prompt["market"],
        collected_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
        text=text.strip(),
        cited_urls=list(dict.fromkeys(clean_url(u) for u in urls)),  # drop duplicates, keep order
        raw=payload,
    )

5. Match brands and domains

The matcher finds every tracked brand and alias in the answer and records the first mention as a rank among the tracked brands, where 1 means named first. Word boundaries stop "Quillvaneian" from counting, and cited URLs are matched by hostname, which accepts subdomains and rejects lookalikes such as notQuillvane.example.

def hostname(source):
    """Host of a cited URL, or the lowercased label when the source is a publisher name."""
    if not source.startswith(("http://", "https://")):
        return source.strip().lower()
    return (urlsplit(source).hostname or "").lower().removeprefix("www.")


def cites(brand, host):
    """True when a cited host is the brand's domain, or a publisher label naming the brand."""
    names = {n.lower() for n in (brand["name"], *brand.get("aliases", []))}
    return host in names or any(domain_matches(host, d) for d in brand["domains"])


def domain_matches(host, domain):
    return host == domain or host.endswith("." + domain)


@lru_cache(maxsize=None)
def name_pattern(names):
    # Longest names first so "Vane Pro" wins over "Vane"; the lookarounds keep a
    # brand from matching inside a longer word.
    alternatives = "|".join(re.escape(n) for n in sorted(names, key=len, reverse=True))
    return re.compile(rf"(?<!\w)(?:{alternatives})(?!\w)", re.IGNORECASE)




def analyze(text, cited_urls, brands):
    """Return (brand, is_own, mentioned, position, cited) for every tracked brand."""
    offsets = {}
    for brand in brands:
        match = name_pattern((brand["name"], *brand.get("aliases", []))).search(text)
        if match:
            offsets[brand["name"]] = match.start()
    rank = {name: i for i, name in enumerate(sorted(offsets, key=offsets.get), start=1)}  # 1 = named first
    hosts = {hostname(u) for u in cited_urls}
    return [
        (
            b["name"],
            int(b["is_own"]),
            int(b["name"] in offsets),
            rank.get(b["name"]),
            int(any(cites(b, h) for h in hosts)),
        )
        for b in brands
    ]

analyze() returns one tuple per brand: its name, whether it's your own, whether it was mentioned, its position, and whether the answer cited its domain. Try it on a made-up answer:

python -c "import llm_tracker as t; b = t.load_config('config.toml')['brands']; print(t.analyze('Ledgerwren is solid, but Quillvane is cheaper.', ['https://www.Quillvane.example/pricing'], b))"

Result:

[('Quillvane', 1, 1, 2, 1), ('Ledgerwren', 0, 1, 1, 0), ('Tarnmere', 0, 0, None, 0)]

Ledgerwren was named first, Quillvane came second and got cited, and Tarnmere never appeared.

6. Store every run

SQLite holds two tables. runs has one row per answer, with the untouched API response in raw_response, and brand_hits has one row per tracked brand per answer:

SCHEMA = """
CREATE TABLE IF NOT EXISTS runs (
    id           INTEGER PRIMARY KEY,
    cycle_id     TEXT NOT NULL,
    collected_at TEXT NOT NULL,
    engine       TEXT NOT NULL,
    prompt_text  TEXT NOT NULL,
    prompt_type  TEXT NOT NULL,
    market       TEXT NOT NULL,
    run_number   INTEGER NOT NULL,
    answer_text  TEXT NOT NULL,
    cited_urls   TEXT NOT NULL,   -- JSON list, tracking parameters removed
    raw_response TEXT NOT NULL    -- untouched API response, kept for re-parsing
);
CREATE TABLE IF NOT EXISTS brand_hits (
    run_id    INTEGER NOT NULL REFERENCES runs(id),
    brand     TEXT NOT NULL,
    is_own    INTEGER NOT NULL,
    mentioned INTEGER NOT NULL,
    position  INTEGER,
    cited     INTEGER NOT NULL,
    PRIMARY KEY (run_id, brand)
);
"""

These functions open the database and save each answer in a single transaction, so a crash never leaves a half-written row:

def connect(path):
    conn = sqlite3.connect(path)
    conn.row_factory = sqlite3.Row
    conn.executescript(SCHEMA)
    return conn




def insert_hits(conn, run_id, hits):
    conn.executemany(
        "INSERT INTO brand_hits (run_id, brand, is_own, mentioned, position, cited) VALUES (?, ?, ?, ?, ?, ?)",
        [(run_id, *hit) for hit in hits],
    )




def save_run(conn, cycle_id, rec, hits):
    with conn:  # one transaction per answer
        cur = conn.execute(
            """INSERT INTO runs (cycle_id, collected_at, engine, prompt_text, prompt_type, market,
                                 run_number, answer_text, cited_urls, raw_response)
               VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
            (cycle_id, rec.collected_at, rec.engine, rec.prompt, rec.prompt_type, rec.market, rec.run,
             rec.text, json.dumps(rec.cited_urls), json.dumps(rec.raw)),
        )
        insert_hits(conn, cur.lastrowid, hits)

Saved answers pay off when you edit an alias. rematch() recomputes every brand hit with no new API calls, and cmd_rematch() is its command:

def rematch(conn, brands):
    """Recompute every brand hit from stored answers, for example after editing aliases."""
    rows = conn.execute("SELECT id, answer_text, cited_urls FROM runs").fetchall()
    with conn:
        conn.execute("DELETE FROM brand_hits")
        for row in rows:
            insert_hits(conn, row["id"], analyze(row["answer_text"], json.loads(row["cited_urls"]), brands))
    return len(rows)




def cmd_rematch(args, cfg):
    conn = connect(args.db)
    print(f"Recomputed brand hits for {rematch(conn, cfg['brands'])} stored answers.")
    conn.close()
    return 0

For more on the database side, see storing data in SQLite. SQLite is enough for a single team, so move to a server database such as PostgreSQL once several people or jobs write at once.

7. Schedule runs

The runner builds one job per prompt, engine, and run, works through them on five threads, and saves each answer as it arrives. A failed task is logged and skipped, so one bad task doesn't sink the cycle:

def build_jobs(cfg):
    runs = cfg["settings"]["runs_per_prompt"]
    return [(p, e, n) for p in cfg["prompts"] for e in cfg["engines"] for n in range(1, runs + 1)]




def collect(job, cfg):
    prompt, engine, run = job
    task_id = submit(engine["target"], prompt["text"], cfg["markets"][prompt["market"]], engine.get("params", {}))
    payload = wait_for_results(task_id, cfg["settings"]["task_timeout_seconds"], cfg["settings"]["poll_interval_seconds"])
    return build_record(engine["name"], engine["target"], prompt, run, payload)




def run_cycle(cfg, conn, log=print):
    """Run every job once. Returns (cycle_id, stored_count, failures)."""
    cycle_id = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    jobs = build_jobs(cfg)
    stored, failures = 0, []
    with ThreadPoolExecutor(max_workers=cfg["settings"]["max_workers"]) as pool:
        futures = {pool.submit(collect, job, cfg): job for job in jobs}
        for future in as_completed(futures):
            prompt, engine, run = futures[future]
            try:
                rec = future.result()
            except Exception as exc:  # one failed task should not sink the whole cycle
                failures.append((engine["name"], run, prompt["text"], str(exc)))
                log(f"FAILED {engine['name']} run {run}: {prompt['text'][:50]} ({exc})")
                continue
            save_run(conn, cycle_id, rec, analyze(rec.text, rec.cited_urls, cfg["brands"]))
            stored += 1
    return cycle_id, stored, failures

Next, the run command and the entry point. Commands share --config and --db:

def cmd_run(args, cfg):
    if args.runs:
        cfg["settings"]["runs_per_prompt"] = args.runs
    jobs = build_jobs(cfg)
    print(
        f"{len(cfg['prompts'])} prompts x {len(cfg['engines'])} engines x "
        f"{cfg['settings']['runs_per_prompt']} runs = {len(jobs)} tasks"
    )
    if args.dry_run:
        return 0
    require_key()
    conn = connect(args.db)
    cycle_id, stored, failures = run_cycle(cfg, conn)
    conn.close()
    print(f"Cycle {cycle_id}: stored {stored} of {len(jobs)} answers, {len(failures)} failed.")
    return 1 if failures else 0


def main(argv=None):
    parser = argparse.ArgumentParser(description="LLM visibility tracker")
    parser.add_argument("--config", default="config.toml")
    parser.add_argument("--db", default="tracker.db")
    sub = parser.add_subparsers(dest="command", required=True)
    run = sub.add_parser("run", help="run one collection cycle")
    run.add_argument("--runs", type=int, help="override runs_per_prompt for this cycle")
    run.add_argument("--dry-run", action="store_true", help="print the task count and exit")
    run.set_defaults(func=cmd_run)
    sub.add_parser("rematch", help="recompute brand hits after editing aliases").set_defaults(func=cmd_rematch)
    args = parser.parse_args(argv)
    return args.func(args, load_config(args.config))




if __name__ == "__main__":
    sys.exit(main())

Check the task count before spending anything:

python llm_tracker.py run --dry-run

Then run:

python llm_tracker.py run 

It will collect the first cycle. It exits with code 1 if any task failed, so a monitored scheduler can flag partial cycles.

A cron job is the simplest scheduler. This entry runs the tracker every Monday at 06:00 server time:

0 6 * * 1 cd /path/to/llm-visibility-tracker && venv/bin/python llm_tracker.py run >> tracker.log 2>&1

Most CI runners start each job with a clean disk, so a CI schedule needs tracker.db stored somewhere persistent or a server database. Our guide on scheduling scraping tasks covers more options.

8. Report

The report prints mention rate, citation rate, and share of voice per engine and per prompt, the change since the previous cycle, and the most cited domains. This step adds code above the entry point, so first delete main() and the two lines under it from the bottom of the file. The final version returns at the end of this step.

Find the latest two cycles and calculate the metrics for any grouping. Share of voice stays empty when no tracked brand was mentioned at all:

BY_ENGINE = ("engine",)
BY_PROMPT = ("prompt_text", "prompt_type", "market", "engine")




def cycle_ids(conn):
    """Return the latest cycle and the one before it."""
    rows = conn.execute("SELECT DISTINCT cycle_id FROM runs ORDER BY cycle_id DESC LIMIT 2").fetchall()
    return (rows[0][0] if rows else None), (rows[1][0] if len(rows) > 1 else None)




def metrics(conn, cycle_id, group_by, prompt_type=None):
    """Mention rate, citation rate, and share of voice for the own brand, per group."""
    if cycle_id is None:
        return {}
    only = " AND r.prompt_type = ?" if prompt_type else ""
    cols = ", ".join(f"r.{c}" for c in group_by)  # group_by comes from the constants above
    rows = conn.execute(
        f"""SELECT {cols},
                   COUNT(DISTINCT r.id) AS runs,
                   SUM(CASE WHEN h.is_own = 1 THEN h.mentioned ELSE 0 END) AS own_mentions,
                   SUM(CASE WHEN h.is_own = 1 THEN h.cited ELSE 0 END) AS own_cited,
                   SUM(h.mentioned) AS all_mentions
            FROM runs r JOIN brand_hits h ON h.run_id = r.id
            WHERE r.cycle_id = ?{only}
            GROUP BY {cols}
            ORDER BY {cols}""",
        (cycle_id, prompt_type) if prompt_type else (cycle_id,),
    ).fetchall()
    return {
        tuple(row[c] for c in group_by): {
            "runs": row["runs"],
            "mention_rate": row["own_mentions"] / row["runs"],
            "citation_rate": row["own_cited"] / row["runs"],
            "share_of_voice": row["own_mentions"] / row["all_mentions"] if row["all_mentions"] else None,
        }
        for row in rows
    }

Most cited domains, counting each domain once per answer:

def top_domains(conn, cycle_id, brands, limit=10):
    """Domains cited most often, counting each domain once per answer."""
    counts = Counter()
    for (urls,) in conn.execute("SELECT cited_urls FROM runs WHERE cycle_id = ?", (cycle_id,)):
        counts.update({h for h in (hostname(u) for u in json.loads(urls)) if h})
    out = []
    for host, n in counts.most_common(limit):
        owner = next((b["name"] for b in brands if cites(b, host)), "")
        out.append((host, n, owner))
    return out

The formatting and the report itself, with changes in percentage points (pp) and a dash where there's nothing to compare:

def pct(value):
    return "-" if value is None else f"{value * 100:.0f}%"




def change(now, before):
    return "-" if now is None or before is None else f"{(now - before) * 100:+.0f} pp"




def report_row(label, m, prev):
    cells = [f"{label:<58} {m['runs']:>4} "]
    for key in ("mention_rate", "citation_rate", "share_of_voice"):
        cells.append(f" {pct(m[key]):>5} {change(m[key], prev and prev[key]):>7} ")
    return "".join(cells)




def build_report(conn, cfg, csv_path=None):
    latest, previous = cycle_ids(conn)
    if latest is None:
        return "No runs stored yet. Run the tracker first."
    own = cfg["brands"][0]["name"]
    lines = [
        f"Cycle {latest}" + (f" (compared with {previous})" if previous else " (no earlier cycle to compare)"),
        f"Brand: {own}. Columns are mention rate, citation rate, and share of voice (voice). Changes are in percentage points (pp).",
    ]
    sections = (("By engine (category prompts only)", BY_ENGINE, "category"), ("By prompt", BY_PROMPT, None))
    for title, group_by, only in sections:
        current, before = metrics(conn, latest, group_by, only), metrics(conn, previous, group_by, only)
        lines += ["", title, f"{'':<58} {'runs':>4}  {'mention':>13}  {'citation':>13}  {'voice':>13}"]
        for key, m in current.items():
            if group_by == BY_ENGINE:
                label = key[0]
            else:
                prompt, ptype, market, engine = key
                label = f"{engine} | {ptype} | {market} | {prompt}"
                label = label if len(label) <= 58 else label[:55] + "..."
            lines.append(report_row(label, m, before.get(key)))
    lines += ["", "Most cited domains (answers citing each)"]
    lines += [f"  {host:<40} {n:>4}  {owner}" for host, n, owner in top_domains(conn, latest, cfg["brands"])]
    if csv_path:
        export_csv(conn, latest, previous, csv_path)
        lines += ["", f"Wrote {csv_path}"]
    return "\n".join(lines)

The CSV export:

def export_csv(conn, latest, previous, path):
    current, before = metrics(conn, latest, BY_PROMPT), metrics(conn, previous, BY_PROMPT)
    rates = ("mention_rate", "citation_rate", "share_of_voice")


    def rounded(x):
        return "" if x is None else round(x, 4)


    with open(path, "w", newline="", encoding="utf-8") as fh:
        writer = csv.writer(fh)
        writer.writerow(["cycle", "engine", "prompt", "type", "market", "runs", *rates, *(f"{r}_change" for r in rates)])
        for (prompt, ptype, market, engine), m in current.items():
            prev = before.get((prompt, ptype, market, engine))
            changes = [
                "" if prev is None or m[r] is None or prev[r] is None else round(m[r] - prev[r], 4) for r in rates
            ]
            writer.writerow([latest, engine, prompt, ptype, market, m["runs"], *(rounded(m[r]) for r in rates), *changes])

Last, the report command and the final main(), which replaces the one you deleted:

def cmd_report(args, cfg):
    conn = connect(args.db)
    print(build_report(conn, cfg, args.csv))
    conn.close()
    return 0


def main(argv=None):
    parser = argparse.ArgumentParser(description="LLM visibility tracker")
    parser.add_argument("--config", default="config.toml")
    parser.add_argument("--db", default="tracker.db")
    sub = parser.add_subparsers(dest="command", required=True)
    run = sub.add_parser("run", help="run one collection cycle")
    run.add_argument("--runs", type=int, help="override runs_per_prompt for this cycle")
    run.add_argument("--dry-run", action="store_true", help="print the task count and exit")
    run.set_defaults(func=cmd_run)
    report = sub.add_parser("report", help="print metrics for the latest cycle")
    report.add_argument("--csv", help="also write the per-prompt metrics to this CSV file")
    report.set_defaults(func=cmd_report)
    sub.add_parser("rematch", help="recompute brand hits after editing aliases").set_defaults(func=cmd_rematch)
    args = parser.parse_args(argv)
    return args.func(args, load_config(args.config))




if __name__ == "__main__":
    sys.exit(main())

Run the report after your first cycle, and again after the next one to see the changes:

python llm_tracker.py report --csv report.csv

That's the whole tracker. One config file, one script. 

Quick start

Here's a quick reference on how to get the script running:

  1. Create the .env file with your Decodo API key (step 1).
  2. Create the config.toml file with the desired configuration (step 2).
  3. Create the llm_tracker.py file with the following code (steps 3-8):
"""LLM visibility tracker: collect AI answers through the Decodo Web Data API and measure brand visibility."""
import argparse
import csv
import json
import os
import re
import sqlite3
import sys
import threading
import time
import tomllib
from collections import Counter
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from datetime import datetime, timezone
from functools import lru_cache
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit




import requests
from dotenv import load_dotenv




def load_config(path):
    with open(path, "rb") as fh:
        cfg = tomllib.load(fh)
    cfg["settings"] = {
        "runs_per_prompt": 3,
        "max_workers": 5,
        "poll_interval_seconds": 5,
        "task_timeout_seconds": 300,
        **cfg.get("settings", {}),
    }
    for prompt in cfg["prompts"]:
        if prompt["market"] not in cfg["markets"]:
            sys.exit(f"Prompt market '{prompt['market']}' is not defined under [markets]")
    # Own brand first, then competitors. Domains are lowercased, without a leading www.
    brands = [{**cfg["brand"], "is_own": True}] + [{**c, "is_own": False} for c in cfg.get("competitors", [])]
    for brand in brands:
        brand["domains"] = [d.lower().removeprefix("www.") for d in brand.get("domains", [])]
    cfg["brands"] = brands
    return cfg




BASE_URL = "https://data.decodo.com/v1"
PROMPT_FIELD = {"google_ai_mode": "query"}  # the other templates take "prompt"






class ApiError(RuntimeError):
    pass






def require_key():
    load_dotenv()
    key = os.environ.get("DECODO_API_KEY", "")
    if not key or key == "YOUR_WEB_DATA_API_KEY":
        sys.exit("Set DECODO_API_KEY in your .env file.")






_local = threading.local()  # one HTTP session per worker thread




def session():
    if not hasattr(_local, "session"):
        _local.session = requests.Session()
        _local.session.headers.update(
            {
                "Authorization": f"Bearer {os.environ['DECODO_API_KEY']}",
                "Accept": "application/json",
                "Content-Type": "application/json",
            }
        )
    return _local.session




def api(method, path, **kwargs):
    """Call the Web Data API, retrying rate limits and server errors."""
    for attempt in range(1, 4):
        last = attempt == 3
        try:
            resp = session().request(method, f"{BASE_URL}{path}", timeout=60, **kwargs)
        except requests.RequestException as exc:
            if last:
                raise ApiError(f"{method} {path} failed: {exc}") from exc
            time.sleep(2**attempt)
            continue
        if resp.status_code in (429, 500, 502, 503, 504) and not last:
            time.sleep(2**attempt)
            continue
        if resp.status_code >= 400:
            raise ApiError(f"{method} {path} returned {resp.status_code}: {resp.text[:200]}")
        return resp.json()




def submit(target, text, geo, params):
    body = {"target": target, PROMPT_FIELD.get(target, "prompt"): text, "parse": True, "geo": geo, **params}
    return str(api("POST", "/task", json=body)["id"])






def wait_for_results(task_id, timeout, interval):
    deadline = time.monotonic() + timeout
    while True:
        status = api("GET", f"/task/{task_id}").get("status")
        if status == "done":
            return api("GET", f"/task/{task_id}/results")
        if status == "faulted":
            item = (api("GET", f"/task/{task_id}/results").get("results") or [{}])[0]
            raise ApiError(f"Task {task_id} faulted (status code {item.get('status_code')})")
        if time.monotonic() >= deadline:
            raise ApiError(f"Task {task_id} still '{status}' after {timeout:.0f}s")
        time.sleep(interval)




TRACKING_PARAMS = {"gclid", "fbclid"}






def clean_url(url):
    """Drop tracking parameters (utm_*, gclid, fbclid) and the fragment."""
    if not url.startswith(("http://", "https://")):
        return url.strip()                                                      
    parts = urlsplit(url.strip())
    query = [
        (k, v)
        for k, v in parse_qsl(parts.query, keep_blank_values=True)
        if not k.lower().startswith("utm_") and k.lower() not in TRACKING_PARAMS
    ]
    return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), ""))




@dataclass
class Record:
    engine: str
    prompt: str
    prompt_type: str
    run: int
    market: str
    collected_at: str
    text: str
    cited_urls: list
    raw: dict






def content_of(payload):
    """Return the dict holding the answer fields.




    The Web Data API pages show `content` holding the fields directly. The older
    ChatGPT page nests them one level deeper under `content.results`, so accept both.
    """
    results = payload.get("results") or []
    if not results:
        raise ValueError("Response has no results")
    content = results[0].get("content") or {}
    inner = content.get("results")
    return inner if isinstance(inner, dict) else content




def from_chatgpt(content):
    urls = [c["url"] for c in content.get("citations") or [] if c.get("url")]
    text = content.get("response_text") or ""
    # With search on, sponsored blocks can trail the answer, followed by ad data that can
    # run to tens of thousands of characters. Cut the text where the first ad block starts.
    marker = text.find("About this ad")
    if marker != -1:
        start = text.rfind("Learn more", 0, marker)
        text = text[: start if start != -1 else marker].rstrip()
    return text, urls






def is_redirect(url):
    parts = urlsplit(url)
    return (parts.hostname or "").removeprefix("www.") == "google.com" and parts.path.startswith("/goto")




def from_google_ai_mode(content):
    # Most sources arrive as google.com/goto redirects, which hide the real site. The
    # "links" list names the publisher behind each one, so keep that label instead.
    links = content.get("links") or []
    publishers = {link["url"]: link.get("publisher") for link in links if link.get("url")}
    urls = [u for citation in content.get("citations") or [] for u in citation.get("urls") or [] if u]
    sources = []
    for url in urls + list(publishers):
        if not is_redirect(url):
            sources.append(url)
        elif publishers.get(url):
            sources.append(publishers[url])
    return content.get("response_text") or "", sources




EXTRACTORS = {"chatgpt": from_chatgpt, "google_ai_mode": from_google_ai_mode}




def build_record(engine_name, target, prompt, run, payload):
    text, urls = EXTRACTORS[target](content_of(payload))
    if not text.strip():
        raise ValueError(f"{engine_name} returned no answer text (is the market supported?)")
    return Record(
        engine=engine_name,
        prompt=prompt["text"],
        prompt_type=prompt["type"],
        run=run,
        market=prompt["market"],
        collected_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
        text=text.strip(),
        cited_urls=list(dict.fromkeys(clean_url(u) for u in urls)),  # drop duplicates, keep order
        raw=payload,
    )




def hostname(source):
    """Host of a cited URL, or the lowercased label when the source is a publisher name."""
    if not source.startswith(("http://", "https://")):
        return source.strip().lower()
    return (urlsplit(source).hostname or "").lower().removeprefix("www.")




def cites(brand, host):
    """True when a cited host is the brand's domain, or a publisher label naming the brand."""
    names = {n.lower() for n in (brand["name"], *brand.get("aliases", []))}
    return host in names or any(domain_matches(host, d) for d in brand["domains"])




def domain_matches(host, domain):
    return host == domain or host.endswith("." + domain)




@lru_cache(maxsize=None)
def name_pattern(names):
    # Longest names first so "Vane Pro" wins over "Vane"; the lookarounds keep a
    # brand from matching inside a longer word.
    alternatives = "|".join(re.escape(n) for n in sorted(names, key=len, reverse=True))
    return re.compile(rf"(?<!\w)(?:{alternatives})(?!\w)", re.IGNORECASE)






def analyze(text, cited_urls, brands):
    """Return (brand, is_own, mentioned, position, cited) for every tracked brand."""
    offsets = {}
    for brand in brands:
        match = name_pattern((brand["name"], *brand.get("aliases", []))).search(text)
        if match:
            offsets[brand["name"]] = match.start()
    rank = {name: i for i, name in enumerate(sorted(offsets, key=offsets.get), start=1)}  # 1 = named first
    hosts = {hostname(u) for u in cited_urls}
    return [
        (
            b["name"],
            int(b["is_own"]),
            int(b["name"] in offsets),
            rank.get(b["name"]),
            int(any(cites(b, h) for h in hosts)),
        )
        for b in brands
    ]




SCHEMA = """
CREATE TABLE IF NOT EXISTS runs (
    id           INTEGER PRIMARY KEY,
    cycle_id     TEXT NOT NULL,
    collected_at TEXT NOT NULL,
    engine       TEXT NOT NULL,
    prompt_text  TEXT NOT NULL,
    prompt_type  TEXT NOT NULL,
    market       TEXT NOT NULL,
    run_number   INTEGER NOT NULL,
    answer_text  TEXT NOT NULL,
    cited_urls   TEXT NOT NULL,   -- JSON list, tracking parameters removed
    raw_response TEXT NOT NULL    -- untouched API response, kept for re-parsing
);
CREATE TABLE IF NOT EXISTS brand_hits (
    run_id    INTEGER NOT NULL REFERENCES runs(id),
    brand     TEXT NOT NULL,
    is_own    INTEGER NOT NULL,
    mentioned INTEGER NOT NULL,
    position  INTEGER,
    cited     INTEGER NOT NULL,
    PRIMARY KEY (run_id, brand)
);
"""
def connect(path):
    conn = sqlite3.connect(path)
    conn.row_factory = sqlite3.Row
    conn.executescript(SCHEMA)
    return conn






def insert_hits(conn, run_id, hits):
    conn.executemany(
        "INSERT INTO brand_hits (run_id, brand, is_own, mentioned, position, cited) VALUES (?, ?, ?, ?, ?, ?)",
        [(run_id, *hit) for hit in hits],
    )






def save_run(conn, cycle_id, rec, hits):
    with conn:  # one transaction per answer
        cur = conn.execute(
            """INSERT INTO runs (cycle_id, collected_at, engine, prompt_text, prompt_type, market,
                                 run_number, answer_text, cited_urls, raw_response)
               VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
            (cycle_id, rec.collected_at, rec.engine, rec.prompt, rec.prompt_type, rec.market, rec.run,
             rec.text, json.dumps(rec.cited_urls), json.dumps(rec.raw)),
        )
        insert_hits(conn, cur.lastrowid, hits)




def rematch(conn, brands):
    """Recompute every brand hit from stored answers, for example after editing aliases."""
    rows = conn.execute("SELECT id, answer_text, cited_urls FROM runs").fetchall()
    with conn:
        conn.execute("DELETE FROM brand_hits")
        for row in rows:
            insert_hits(conn, row["id"], analyze(row["answer_text"], json.loads(row["cited_urls"]), brands))
    return len(rows)






def cmd_rematch(args, cfg):
    conn = connect(args.db)
    print(f"Recomputed brand hits for {rematch(conn, cfg['brands'])} stored answers.")
    conn.close()
    return 0




def build_jobs(cfg):
    runs = cfg["settings"]["runs_per_prompt"]
    return [(p, e, n) for p in cfg["prompts"] for e in cfg["engines"] for n in range(1, runs + 1)]






def collect(job, cfg):
    prompt, engine, run = job
    task_id = submit(engine["target"], prompt["text"], cfg["markets"][prompt["market"]], engine.get("params", {}))
    payload = wait_for_results(task_id, cfg["settings"]["task_timeout_seconds"], cfg["settings"]["poll_interval_seconds"])
    return build_record(engine["name"], engine["target"], prompt, run, payload)






def run_cycle(cfg, conn, log=print):
    """Run every job once. Returns (cycle_id, stored_count, failures)."""
    cycle_id = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
    jobs = build_jobs(cfg)
    stored, failures = 0, []
    with ThreadPoolExecutor(max_workers=cfg["settings"]["max_workers"]) as pool:
        futures = {pool.submit(collect, job, cfg): job for job in jobs}
        for future in as_completed(futures):
            prompt, engine, run = futures[future]
            try:
                rec = future.result()
            except Exception as exc:  # one failed task should not sink the whole cycle
                failures.append((engine["name"], run, prompt["text"], str(exc)))
                log(f"FAILED {engine['name']} run {run}: {prompt['text'][:50]} ({exc})")
                continue
            save_run(conn, cycle_id, rec, analyze(rec.text, rec.cited_urls, cfg["brands"]))
            stored += 1
    return cycle_id, stored, failures




def cmd_run(args, cfg):
    if args.runs:
        cfg["settings"]["runs_per_prompt"] = args.runs
    jobs = build_jobs(cfg)
    print(
        f"{len(cfg['prompts'])} prompts x {len(cfg['engines'])} engines x "
        f"{cfg['settings']['runs_per_prompt']} runs = {len(jobs)} tasks"
    )
    if args.dry_run:
        return 0
    require_key()
    conn = connect(args.db)
    cycle_id, stored, failures = run_cycle(cfg, conn)
    conn.close()
    print(f"Cycle {cycle_id}: stored {stored} of {len(jobs)} answers, {len(failures)} failed.")
    return 1 if failures else 0






BY_ENGINE = ("engine",)
BY_PROMPT = ("prompt_text", "prompt_type", "market", "engine")






def cycle_ids(conn):
    """Return the latest cycle and the one before it."""
    rows = conn.execute("SELECT DISTINCT cycle_id FROM runs ORDER BY cycle_id DESC LIMIT 2").fetchall()
    return (rows[0][0] if rows else None), (rows[1][0] if len(rows) > 1 else None)






def metrics(conn, cycle_id, group_by, prompt_type=None):
    """Mention rate, citation rate, and share of voice for the own brand, per group."""
    if cycle_id is None:
        return {}
    only = " AND r.prompt_type = ?" if prompt_type else ""
    cols = ", ".join(f"r.{c}" for c in group_by)  # group_by comes from the constants above
    rows = conn.execute(
        f"""SELECT {cols},
                   COUNT(DISTINCT r.id) AS runs,
                   SUM(CASE WHEN h.is_own = 1 THEN h.mentioned ELSE 0 END) AS own_mentions,
                   SUM(CASE WHEN h.is_own = 1 THEN h.cited ELSE 0 END) AS own_cited,
                   SUM(h.mentioned) AS all_mentions
            FROM runs r JOIN brand_hits h ON h.run_id = r.id
            WHERE r.cycle_id = ?{only}
            GROUP BY {cols}
            ORDER BY {cols}""",
        (cycle_id, prompt_type) if prompt_type else (cycle_id,),
    ).fetchall()
    return {
        tuple(row[c] for c in group_by): {
            "runs": row["runs"],
            "mention_rate": row["own_mentions"] / row["runs"],
            "citation_rate": row["own_cited"] / row["runs"],
            "share_of_voice": row["own_mentions"] / row["all_mentions"] if row["all_mentions"] else None,
        }
        for row in rows
    }




def top_domains(conn, cycle_id, brands, limit=10):
    """Domains cited most often, counting each domain once per answer."""
    counts = Counter()
    for (urls,) in conn.execute("SELECT cited_urls FROM runs WHERE cycle_id = ?", (cycle_id,)):
        counts.update({h for h in (hostname(u) for u in json.loads(urls)) if h})
    out = []
    for host, n in counts.most_common(limit):
        owner = next((b["name"] for b in brands if cites(b, host)), "")
        out.append((host, n, owner))
    return out




def pct(value):
    return "-" if value is None else f"{value * 100:.0f}%"






def change(now, before):
    return "-" if now is None or before is None else f"{(now - before) * 100:+.0f} pp"






def report_row(label, m, prev):
    cells = [f"{label:<58} {m['runs']:>4} "]
    for key in ("mention_rate", "citation_rate", "share_of_voice"):
        cells.append(f" {pct(m[key]):>5} {change(m[key], prev and prev[key]):>7} ")
    return "".join(cells)






def build_report(conn, cfg, csv_path=None):
    latest, previous = cycle_ids(conn)
    if latest is None:
        return "No runs stored yet. Run the tracker first."
    own = cfg["brands"][0]["name"]
    lines = [
        f"Cycle {latest}" + (f" (compared with {previous})" if previous else " (no earlier cycle to compare)"),
        f"Brand: {own}. Columns are mention rate, citation rate, and share of voice (voice). Changes are in percentage points (pp).",
    ]
    sections = (("By engine (category prompts only)", BY_ENGINE, "category"), ("By prompt", BY_PROMPT, None))
    for title, group_by, only in sections:
        current, before = metrics(conn, latest, group_by, only), metrics(conn, previous, group_by, only)
        lines += ["", title, f"{'':<58} {'runs':>4}  {'mention':>13}  {'citation':>13}  {'voice':>13}"]
        for key, m in current.items():
            if group_by == BY_ENGINE:
                label = key[0]
            else:
                prompt, ptype, market, engine = key
                label = f"{engine} | {ptype} | {market} | {prompt}"
                label = label if len(label) <= 58 else label[:55] + "..."
            lines.append(report_row(label, m, before.get(key)))
    lines += ["", "Most cited domains (answers citing each)"]
    lines += [f"  {host:<40} {n:>4}  {owner}" for host, n, owner in top_domains(conn, latest, cfg["brands"])]
    if csv_path:
        export_csv(conn, latest, previous, csv_path)
        lines += ["", f"Wrote {csv_path}"]
    return "\n".join(lines)




def export_csv(conn, latest, previous, path):
    current, before = metrics(conn, latest, BY_PROMPT), metrics(conn, previous, BY_PROMPT)
    rates = ("mention_rate", "citation_rate", "share_of_voice")




    def rounded(x):
        return "" if x is None else round(x, 4)




    with open(path, "w", newline="", encoding="utf-8") as fh:
        writer = csv.writer(fh)
        writer.writerow(["cycle", "engine", "prompt", "type", "market", "runs", *rates, *(f"{r}_change" for r in rates)])
        for (prompt, ptype, market, engine), m in current.items():
            prev = before.get((prompt, ptype, market, engine))
            changes = [
                "" if prev is None or m[r] is None or prev[r] is None else round(m[r] - prev[r], 4) for r in rates
            ]
            writer.writerow([latest, engine, prompt, ptype, market, m["runs"], *(rounded(m[r]) for r in rates), *changes])




def cmd_report(args, cfg):
    conn = connect(args.db)
    print(build_report(conn, cfg, args.csv))
    conn.close()
    return 0




def main(argv=None):
    parser = argparse.ArgumentParser(description="LLM visibility tracker")
    parser.add_argument("--config", default="config.toml")
    parser.add_argument("--db", default="tracker.db")
    sub = parser.add_subparsers(dest="command", required=True)
    run = sub.add_parser("run", help="run one collection cycle")
    run.add_argument("--runs", type=int, help="override runs_per_prompt for this cycle")
    run.add_argument("--dry-run", action="store_true", help="print the task count and exit")
    run.set_defaults(func=cmd_run)
    report = sub.add_parser("report", help="print metrics for the latest cycle")
    report.add_argument("--csv", help="also write the per-prompt metrics to this CSV file")
    report.set_defaults(func=cmd_report)
    sub.add_parser("rematch", help="recompute brand hits after editing aliases").set_defaults(func=cmd_rematch)
    args = parser.parse_args(argv)
    return args.func(args, load_config(args.config))






if __name__ == "__main__":
    sys.exit(main())    

  4. Run it with:

python llm_tracker.py run

  5. Generate the report with:

python llm_tracker.py report --csv report.csv

How to read the results

Start with the prompt type, not the totals. Brand and comparison prompts name your brand, so the engines repeat it and the mention rate sits near 100% no matter what you do. Category prompts name nobody, which makes them the honest test of visibility. Read those rows first.

Then compare the three metrics. Say a category prompt on ChatGPT shows these numbers (invented, for illustration):

  • Mention rate. 33%, so Quillvane appears in one answer out of three.
  • Citation rate. 8%, so it's named far more often than its own site is cited.
  • Share of voice. 20%, so when any tracked brand is named, one in five mentions is yours.

A wide gap between mentions and citations means the engines know the name but source their claims elsewhere. The most cited domains list shows where. Review sites, forums, and comparison pages that keep showing up there are your outreach and content targets.

Treat changes with some restraint. With a few runs per prompt, a swing of several percentage points between two cycles can be plain variance. Act on a shift only when it holds across several cycles or appears across prompts and engines at once.

Position adds a closer read. Being named first is probably better than being named fourth, but that's an assumption, not something the tracker measures.

Build your own tracker or buy an LLM visibility tool?

Build when you need control over prompts, engines, markets, and the raw answers. Buy when nobody on the team can maintain code, or when a client wants a dashboard by Friday.

Building gives you:

  • Your own prompt set and run counts. Nothing is sampled on your behalf.
  • Every raw answer stored, so you can re-parse it when your matching rules change.
  • Any market the templates support, and the freedom to add engines.
  • Costs that scale with API calls instead of a flat plan.

Buying gives you dashboards, alerts, and reports without engineering time, usually priced per prompt, project, or seat. The trade is that prompt sets, run counts, and markets typically follow the vendor's plan, and you see the numbers the vendor chose to show.

To compare the two honestly, reuse the formula from the build section. Monthly tasks equal prompts times engines times runs per prompt times cycles per month, and the sampling section's example works out to 3,250 tasks a month at weekly cadence. Price that figure on the Web Data API pricing page, then add the hours someone will spend keeping the code alive.

If the total still beats a vendor quote, build. If the maintenance line made you wince, that's an answer too.

Collecting AI answers at scale

The hard part of an LLM visibility tracker isn't the matching or the SQL. It's collecting answers reliably from interfaces built for people, on a steady cadence and in several markets.

The Web Data API handles that part of this build. Each engine is a template that takes a prompt and a market, and returns parsed JSON, with rendering and anti-bot handling built in. The tracker only deals with fields such as response_text and citations, and adding a market means adding one line to config.toml. The sources an answer cites appear next to its text, which is what the citation rate is built on. For ChatGPT, the search option turns on web search, and citations are returned when the answer cites sources. Tasks run asynchronously, so the tracker keeps several in flight at once instead of waiting on one answer at a time. That leaves your code focused on analysis rather than on collection.

Bottom line

Tracking LLM visibility comes down to sampling. Run the same prompts repeatedly on each engine, store every answer, and measure mention rate, citation rate, and share of voice across the runs instead of trusting any single reply. The script in this guide does that with one file and one config. Next, swap in your own brand, competitors, and 20 to 50 prompts, run a cycle with three runs each, and read the category rows first.

Compare answers by market

Run the same prompt on the Decodo Web Data API across countries with the geo parameter to see how AI answers change from market to market.

Share article:

About the author

Zilvinas Tamulis

Technical Copywriter

A technical writer with over 4 years of experience, Žilvinas blends his studies in Multimedia & Computer Design with practical expertise in creating user manuals, guides, and technical documentation. His work includes developing web projects used by hundreds daily, drawing from hands-on experience with JavaScript, PHP, and Python.

Connect with Žilvinas via LinkedIn

All information on Decodo Blog is provided on an as is basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Decodo Blog or any third-party websites that may belinked therein.

Frequently asked questions

How many prompts does an LLM visibility tracker need?

Start with 20 to 50 prompts that cover brand, category, and comparison questions, run each one several times per cycle, and add more over time. Search Engine Land suggests 250 to 500 high-intent queries for a stable read at scale.

Does ChatGPT cite sources without web search?

ChatGPT cites sources when it searches the web, and with Decodo's ChatGPT template, that means setting search: true. The template returns citations when the answer cites sources. Answers without search show what the model already knows, so track them as a separate series.

How often should you run an LLM visibility tracker?

Weekly runs are enough for trend lines, while daily runs suit launches or reputation issues and cost about seven times as much. At 50 prompts, 2 engines, and 5 runs per prompt, a weekly cycle takes about 2,170 tasks a month and a daily cycle about 15,000.

How To Build a Rank Tracker

How To Build a Rank Tracker: Manual Checks, Python Automation, and Modern SERP Tracking

On a recent run, wired.com ranked no. 1 on US desktop and no. 2 on UK desktop computers for "best laptop 2026". Same query, same hour, different country. That gap is what a single-number rank tracker misses, especially now that modern SERPs add AI Overview citations, featured snippets, and "People also ask" blocks that most older tracking tools ignore. This post walks through building a tracker that captures all of it, starting with a manual baseline for ground truth, then moving to a Python implementation against a SERP API, and finally setting up a scaling path for more keywords and locations.

neon bug icon glowing inside a rounded square, centered on a dark dotted tech background

How to Automate Web Scraping Tasks: Schedule Your Data Collection with Python, Cron, and Cloud Tools

Web scraping becomes truly valuable when it is automated. It allows you to track competitor prices, monitor job listings, and continuously feed fresh data into AI pipelines. But while building a scraper that works can be exciting, real-world use cases require repeatedly and reliably collecting data at scale, which makes manual or one-off scraping ineffective. 


Scheduling enables this by ensuring consistent execution, reducing errors, and creating reliable data pipelines. In this guide, you will learn how to automate scraping using 3 approaches: in-script scheduling with Python libraries, system-level tools like cron or Task Scheduler, and cloud-based solutions such as GitHub Actions.

JSON 'results' with 'status_code': 200 and 'url': 'https://example.com' on scraping UI showing 'Start scraping'

How to Scrape Google AI Mode: Methods, Tools, and Best Practices

Google AI Mode was launched as a Search Labs experiment in March 2025. It's powered by Gemini 2.5, which synthesizes answers from multiple sources and allows you to ask follow-up questions. Google AI Mode isn't the same as Google search results; it's an entirely full-page conversational interface using different URL parameters, rendering pipelines, and scraping logic. This guide provides a walkthrough of two different approaches: a working Playwright script you can execute right away, and the Decodo Web Data API for production.

© 2018-2026 decodo.com (formerly smartproxy.com). All Rights Reserved