Agentic Compatibility Report: Scoring AI Tools on Real Tasks

TL;DR

• Only 2 of 45 tools, from Amazon and OpenAI, let an agent complete a purchase and pause to check with you first.
• The tool with the best documentation score got stuck on complex layouts, looped, and couldn't beat CAPTCHAs once we tested it ourselves.
• Devin's own demo showed it fixing bugs that didn't exist, and in real testing it finished only 2 of 10 tasks.
• How carefully a tool explains its memory predicts its safety habits far better than how good it's at completing purchases.

What the Agentic Compatibility Report measures

The report scores 45 AI browsing tools on two layers. The first layer is a documented score, based on how clearly each vendor describes a capability across its public docs, pricing pages, and changelogs. The second layer is an adjusted score, which takes those documented claims and checks them against our own hands-on testing.

The 10 capabilities are:

• Form filling. Basic autofill versus arbitrary or complex form completion.
• Transactional actions. Completing a purchase or checkout, not just adding to a cart.
• Multi-step task completion. Chaining actions toward one goal without re-prompting.
• Cross-tab awareness. Reading or referencing content across more than one open tab.
• Session memory. Retaining context within a single browsing session.
• Persistent memory. Retaining context across separate sessions or days.
• Cross-site execution. Acting across more than one domain within a single task.
• Confirmation before irreversible actions. Asking before payment or submit-type actions.
• Scheduled or background tasks. Running without an active user session.
• Third-party integrations. Named connectors beyond the browser itself.

The documented score works like this:

• A score of 0 means the capability was never mentioned.
• A score of 1 means the claim was partial or implied, or contradicted elsewhere on the same site.
• A score of 2 means the vendor stated the capability in clear, direct language.
• The maximum documented score for any tool is 20.

The second evaluation happens with hands-on testing against real workflows. Our team logged what actually happened: crashes, freezes, timeouts, session drops, and outright failures on the exact task each capability claims to cover. Each finding got sorted into one of four gap types, and each gap type carries a different penalty against the documented score:

• Contradicts claim. The capability failed outright on the task it claims to cover; capability score drops to 0.
• Credibility gap. The evidence behind the claim doesn't hold up, such as a demo shown fixing bugs that didn't exist; capability score drops to 0.
• Reliability gap. The capability works but breaks down under real conditions, such as crashes, freezes, or dropped sessions; capability score drops by 1 point.
• Corroborates claim. Real-world testing matched the documented score; no change.

The full list of all 45 tools

The report covers browser extensions, standalone automation platforms, cloud browser infrastructure, and coding agents with a built-in browser tool. Scores range from 18 down to 2, with some totals including a real-world adjustment from our hands-on testing, explained later in the report.

Claude for Chrome and Amazon Nova Act share the top spot with 18 points out of 20. Fellou and ChatGPT Desktop tie for third at 17. Fellou's total already reflects a 2-point deduction from our hands-on testing, more on that below, which is why it no longer sits alone at the top the way its documentation alone would suggest. Skyvern, Vercel Agent Browser, and a four-way tie at 14 points round out the top ten.

No tool scored a perfect 20. Every top performer had at least one gap, and in most cases that gap sat in the same two categories: transactional actions or scheduled tasks. That's not an accident. Those are exactly the spots where a mistake costs real money.

Claude for Chrome lost a point on session memory. Its own docs blend in-session and cross-session claims, with no clean separation between the two. Amazon Nova Act lost a point on cross-tab awareness. Fellou's total includes a real-world penalty, and it still nearly matches them.

The spread shows how uneven this market still is. A few tools document nearly every capability, while many others cover just two or three and stay quiet on the rest.

Checkout is the one task almost everyone avoids

Transactional actions scored 0.43 out of 2, the lowest average in the report. Liability drives that gap. Vendors are wary of an agent buying the wrong thing with someone else's money.

Only six tools scored full marks here: Amazon Nova Act, BrowserOS, ChatGPT Desktop, Minded, Sigma Browser, and Skyvern.

This gap tracks with a bigger shift happening in payments right now. OpenAI and Stripe built the Agentic Commerce Protocol, which lets an AI agent complete a purchase for a shopper. Stripe describes the protocol as the system that already powers Instant Checkout inside ChatGPT. That claim comes from Stripe's own launch announcement. Google has since introduced a competing cart standard. Retail partners for that standard include Target, Walmart, and Wayfair.

Some of the caution comes down to liability. A wrong click on a form field is easy to undo. A wrong click on a payment button isn't, so vendors are far more comfortable publishing claims about the first than the second.

Gabriele Vitke, Product Marketing Team Lead at Decodo, notes, "It isn't a bad thing that some of the agentic tools aren't capable to complete the purchase. With the growing skepticism around safety of AI tools, complaints around the fact that the agent has purchased the wrong product wouldn't help this case. It's better to improve the flows of the tools and only then roll out such features."

Confirmation before irreversible actions splits the field

Ten tools in the report document a pause step. That step asks the user before the agent acts on something irreversible. Those tools are Amazon Nova Act, ChatGPT Chrome Extension, ChatGPT Desktop, Claude for Chrome, Devin, Fellou, Lindy, Steel, and Vercel Agent Browser.

Claude for Chrome's documentation describes a dedicated safety classifier that reviews actions before they run. The docs state plainly that purchases and financial actions wait for the user.

A tool that can complete a purchase and pause for confirmation sits in a small group. ChatGPT Desktop and Amazon Nova Act belong to that group. Most tools that can complete a transaction don't document a confirmation step at all. Full checkout access with no documented safety brake is the riskiest pattern in the whole report. Check for it before you hand an agent your payment details.

51% of agents can chain a task with no brakes

This number might worry regulators more than any single vendor. Of the 35 tools that fully document multi-step task completion, 18 of them, 51%, document zero confirmation step before an irreversible action. Over half the field can already chain a task from start to finish. Fewer than half say they'll stop and ask first.

Scheduled and background tasks make the picture worse. Seventeen tools document the ability to run without an active user session. Seven of those, Airtop, Axiom AI, BrowserOS, Browserbase, HARPA AI, OpenHands, and Opera Neon, pair that with zero documented confirmation step. That means an agent can act while a person sleeps, meets, or simply isn't looking. The vendor's own documentation never mentions a check-in moment.

What happens when the documentation meets real use

Documentation is one layer of evidence. However, it isn't the only one. We ran our own hands-on tests against the highest-stakes claims. Once the testing concluded, some of the agentic tools had lost points:

The report's own top scorer on paper took the hardest hit. Fellou's documentation claimed 19 out of 20 points, full marks on multi-step task completion. In our testing, it got stuck on complex layouts, looped at times, and couldn't get past CAPTCHAs. That capability drops from 2 to 0 under our adjustment rules, taking Fellou's total from 19 to 17, the number shown in the tables above.

Devin fared worse. Its docs support a full score on multi-step task completion. We tested Devin on 10 real tasks. It completed only 2 without hitting an error, a 20% success rate. We also asked it to fix real issues in a GitHub repository directly, and it couldn't. Separately, Devin was shown fixing bugs and files that didn't actually exist in the referenced repository during its own promotional demo, a sign the demoed task chain was staged. Both findings zero out under our rules, multi-step task completion and confirmation before irreversible actions, dropping Devin's total from 13 to 9.

Comet, Perplexity's browser, claims partial support for transactional actions. In our testing, it reported being unable to connect to booking websites at all. That's a direct miss on the exact task type this capability covers, so the score drops from 1 to 0. We also hit a freezing, laggy sidebar during agentic tasks, which knocked its multi-step task completion score down too. Comet's adjusted total falls from 6 to 4.

Manus tells a split story. Its 0 score on transactional actions held up in our testing. Real attempts at booking a restaurant table and online shopping failed, matching what the documentation never claimed in the first place, so that score doesn't move. But its full score on multi-step task completion didn't hold up. We hit frequent crashes and glitches, with Manus struggling mid-chain on the same everyday tasks its docs describe handling. Its session memory score took a reliability penalty too. Manus falls from 5 to 2.

Browser Use, an open-source automation tool, picked up two separate contradictions plus a reliability finding in our testing. The agent failed outright on a second task when we chained tasks sequentially in the same browser session, directly contradicting the claim of chaining without re-prompting. It also launched blank Chrome windows that never navigated, even on the simplest single-step task. A third failure mode, schema validation errors halting execution mid-task, wasn't even recorded in the run history, reinforcing the same multi-step penalty. Browser Use falls from 14 to 10 points.

None of this erases what the documented scores capture. It adds a second layer on top. Some agentic tools described the capabilities accurately, yet still shipped a product that fails at it under real conditions.

"Even though the documentation looked great for most of the tools, our actual tests told a different story. For anyone evaluating tools by documentation alone, that's a good reminder that free trials exist, and they can help you find a tool that actually maximizes your potential, instead of wasting your time pressing the retry button," Gabriele Vitke noted.

Cross-tab awareness belongs to browser extensions

Cross-tab awareness averaged 0.66 out of 2, one of the lowest scores in the report. The tools that scored full marks were almost all browser native extensions. That list includes Claude for Chrome, ChatGPT Chrome Extension, Chrome Gemini, Sigma Browser, Opera Aria, Opera Neon, and Brave Leo.

Standalone automation platforms scored much lower on this capability. Cloud browser infrastructure did too. Browserbase, Airtop, Devin, and Browser Use each scored 0 or 1. Tabs are a concept built into a desktop browser window. A cloud agent driving a headless browser session often has no equivalent concept to point to. That structural gap is unlikely to close soon. It's baked into how these platforms are built.

Why vendors move slowly on full autonomy

The caution around checkout and confirmation steps isn't overblown. It lines up with active security research on browser agents, and the findings are unsettling. Anthropic published research on prompt injection attacks against Claude for Chrome, showing how a malicious webpage can embed hidden instructions that hijack an agent mid-task. A helpful assistant becomes an attacker's tool, and the user never notices.

OpenAI documented a similar risk for ChatGPT Atlas. Its writeup describes an attacker who hides instructions in an email, with the goal of hijacking an agent into forwarding private documents. That scenario appears in OpenAI's own hardening writeup.

Brave's security team found an even more alarming issue in Perplexity's Comet browser. Hidden text on a webpage triggered the browser to act on its own. In one case, it fetched one-time passwords straight from an inbox. That finding was published in Brave's own security research.

Every webpage an agent visits is a potential trap. A payment form or a submit button is exactly where a hijacked agent can do damage that can't be undone. The low scores on transactional actions and confirmation steps aren't a coincidence. They're the industry admitting, quietly, that it doesn't fully trust its own agents yet.

What this means for anyone choosing an agent

A high total score doesn't tell you which agent will actually save your team time. That's the trap with any leaderboard. It rewards breadth, while most businesses only care about depth in one or two categories. A company automating expense report processing doesn't need a tool that scores well across all ten capabilities. It needs one that nails form filling and session memory, and doesn't crash halfway through a batch of receipts.

Start by mapping your workflow to the categories that actually matter for it:

• Back-office data entry and form processing. Weight form filling and reliability under real conditions over anything else.
• Scheduling, monitoring, or recurring reports. Weight scheduled tasks alongside confirmation steps, since an agent running unsupervised needs a way to flag problems rather than fail silently.
• Procurement or purchasing workflows. Weight transactional actions and confirmation before irreversible actions above all else, given how few tools handle both well.
• Research or multi-source lookups. Weight cross-tab awareness and multi-step completion, especially if the work spans several open tabs or sites.

"Someone on your team will still have to own what happens when the agent gets it wrong. An agent buying the wrong inventory or emailing the wrong client won't announce itself. And don't assume the most capable-sounding tool is the cheapest one to deploy. A tool that fails and needs constant babysitting costs more in staff time than a narrower tool that does one thing reliably. Match the tool to the job, not to the longest feature list," Gabriele Vitke noted.

Bottom line

The Agentic Compatibility Report shows an industry that's mostly solved two easy problems: chaining actions and connecting to other tools. It hasn't solved the hard one, knowing when to stop and ask. Only two tools document both a completed transaction and a confirmation step: ChatGPT Desktop and Amazon Nova Act, from OpenAI and Amazon.

While we're seeing the rapid development of agentic tools, we're still left with the same problems users faced at the very beginning of this technology: trust, reliability, and knowing when a tool is actually ready to act on someone's behalf.

Standards like the Agentic Commerce Protocol and Google's cart standard suggest the payment side of this problem is getting real attention, but a shared standard for confirmation and oversight hasn't emerged yet. Until it does, every vendor is left to define "safe enough" on its own, and that inconsistency is exactly what this report captures.