Editorial Team
7 min read
Visa's new benchmark finds that AI shopping models often miss customer requirements and struggle to repeat successful decisions. The tests cover product discovery, leaving checkout and payment reliability for future rounds.
Visa, the digital payments network, says AI models passed just 57.2% of runs in its new simulated-shopping benchmark. The shortfall comes before checkout: assistants must discover what a customer needs and support the right buying decision. Visa is testing that capability while building payment tools for the same emerging market.
Visa’s illustration for its company-reported Agentic Commerce Benchmark, October 2026 snapshot. Source: Visa Payments Frontier.
Visa launched its Agentic Commerce Benchmark on November 5. The company connects consumers, merchants and financial institutions across more than 200 countries and territories. Its research tested 22 AI models from 12 labs on 15 scenarios involving televisions, laptops and personal-care appliances, according to its announcement.
Each model attempted each scenario three times: 45 runs per model, or 990 across the test. The published results are an October 2026 snapshot of selected underlying models, not a test of every finished shopping application or actual consumer transactions. A model supplies the AI reasoning; a deployed assistant also depends on its tools and the shopping environment.
In the published experiment, one AI model plays the shopper and another acts as the shopping assistant. Every assistant searches the same simulated market. The shopper initially reveals only some requirements; the assistant has to elicit the rest through questions or requests for further checks.
A pass requires both discovering every task-defined requirement and reaching the correct decision without misleading the shopper. Selecting a suitable product by luck does not pass. Declining to buy can pass when no offer fits.
One television scenario illustrates the distinction. An opening request identifies a screen size and a need for returns, but further conversation must establish the budget and family members' requirements for video formats and gaming. An assistant that satisfies the opening request alone can still miss the purchase's purpose.
Visa says the five models that uncovered the most hidden buying factors averaged a 70.2% pass rate, against 44.4% for the five that uncovered the fewest. That association points toward asking better questions as an important capability; it does not establish that adding questions alone would produce the same improvement.
Magica chart of Visa’s company-reported simulated shopping outcomes. Purchases could fail because requirements were unmet or not fully discovered; no real payment was tested. Source: Visa Payments Frontier.
The consequential failure is not always refusing to help. Visa's results classify 23.1% of all runs as purchases that did not pass. Those account for roughly 54% of failed runs, a finding also described in the accompanying coverage.
That category includes both buying an unsuitable product and buying without establishing every requirement. It should not be read as evidence that every such selection was the wrong product, or that real money changed hands.
Another 19.7% of runs ended with other unsuccessful outcomes: declining despite an available fit, or reaching no decision, including because of tool-call limits or invalid response formats. Across all models and scenarios, only 34.2% of the three-run sets passed every time. A single successful demonstration therefore gives an incomplete picture of repeatability.
The evaluation also filters some operational failures. Provider or network errors were rerun and excluded. Nine runs stopped by output or session limits that had not been disclosed to the model were replaced with one fresh attempt, whose result counted: six for Muse Wonder and three for MiniMax. Those rules matter when interpreting the figures as evidence about a deployed service's reliability.
Magica chart of Visa’s company-reported cost-group averages. The association does not establish that spending more causes better results. Source: Visa.
Among the AI models tested, Google's Gemini 4 Argon and Meta's Muse Wonder tied at 82.2%, followed by Anthropic's Claude Fable 5.1 at 75.6%, according to Visa's results and methods. The lowest score was 28.9% for Thinking Machines' Inkling.
But Visa cautions that no model clearly leads. Each of the two highest pass rates has a 95% confidence interval of 69% to 91%, overlapping the intervals of the next eleven models. Forty-five runs per model support a comparison on these tasks, not a definitive ranking of general shopping ability.
The five most expensive models per run averaged a 67.1% pass rate, compared with 50.2% for the five least expensive. Yet the individual results resist a simple price rule:
| Model | Pass rate | Model cost per run |
|---|---|---|
| Gemini 4 Argon | 82.2% | $0.54 |
| Muse Wonder | 82.2% | $1.55 |
| Claude Fable 5.1 | 75.6% | $1.82 |
These are provider-reported costs for the model being tested, rather than the full expense of operating a consumer shopping service. The two joint leaders achieved the same observed pass rate at substantially different costs, while the costliest model in Visa's table scored below both. The higher-cost group did better on average in this sample; the results do not show that raising an assistant's budget causes its performance to improve.
Visa brings payments experience to the test and has a commercial stake in agents becoming trusted buyers. It introduced Visa Intelligent Commerce in April 2025, opening its payment network to developers building AI agents. Visa said the initiative built on more than 30 years of AI and machine learning work in risk and fraud. Its announcement described APIs and a commercial partner program, with tokenized payment credentials, consumer-set spending limits and transaction controls. Visa also named collaborators including Anthropic, Microsoft, OpenAI and Stripe. Some providers whose models it now benchmarks are therefore also participants in its commerce initiative.
The company describes the new benchmark as complementary research, rather than directly part of that portfolio of products, programs and partnerships. Its experience with payment security provides relevant infrastructure expertise; the shopping experiment tests a different capability, understanding a buyer's intent.
Visa is also not the only payment network preparing for delegated shopping. Reporting accompanying the study describes both Visa and Mastercard building tools and educating merchants, while identifying unresolved questions about responsibility for rogue purchases or fraud.
The controlled market also leaves out obstacles that Visa identifies elsewhere on the same research site. Its separate Agent Score service assesses merchant websites for agent readiness; Visa describes inconsistent product data, bot-identification problems, human-oriented forms and CAPTCHA barriers. Those are additional deployment constraints, distinct from whether a model can discover a simulated shopper's requirements.
Nor is inconsistent agent behavior a newly discovered research problem. The 2024 tau-bench paper tested agents interacting with simulated users through domain-specific tools and rules, and assessed reliability over repeated trials. Its tasks and models differ from Visa's, so the results cannot establish a performance trend. The precedent narrows Visa's novelty claim: Visa specifically claims a first for requiring discovery of every task-defined customer requirement alongside a correct purchase or no-purchase decision.
Visa plans to expand the benchmark to more scenarios and product categories, then checkout, payment and order placement. It has shared the benchmark with leading model providers for feedback and made results, replays and an interactive demonstration publicly accessible.
The distinction matters for trust. Visa says its September 2026 Trust Index found that 72% of U.S. consumers had used an AI assistant, but only 23% trusted AI to handle payments on their behalf. Those company-reported survey figures measure different behaviors and attitudes; using an assistant does not imply authorizing it to spend.
For developers choosing a model, the immediate question is whether these results hold across more tasks and repeated attempts, at an acceptable operating cost. For consumers and merchants delegating transactions, the next test is whether an assistant carries the agreed requirements through checkout and payment—and how errors are handled. Visa's first round supplies evidence about the buying decision; its planned extensions must establish reliability through the rest of the purchase.
The Cambridge startup has secured backing described as eight figures in capital and compute, including UK government support. Its wager is that cheaper, accurate quantum calculations can supply better AI training data; full performance results remain forthcoming.
Japan’s Nikkei fell 7.09% in Friday morning trading, with AI, chip and data center-related shares hit hard after Wall Street losses. Reports linked the rout to fears of U.S. regulation following Democratic election victories, without identifying an enacted restriction or a cut in AI spending.
Potential investors are pressing Anthropic for a lower valuation as a technology selloff threatens its November IPO timetable. Rapid revenue growth and substantial cash complicate a financing debate driven by losses and long-term infrastructure commitments.
Agility Robotics began trading after its Churchill Capital Corp XI merger, bringing a commercially deployed humanoid robot maker to U.S. public markets. Its next test is turning conditional Digit 5 orders into deliveries and sustainable economics.
Goldman Sachs sees temporary halts potentially affecting about 45% of planned US data center expansion after Democratic election victories. In Texas, where approvals were already paused, the next fight is over how long restrictions last and what developers must pay for.
Core Scientific’s third-quarter colocation revenue rose 57.6% to $215.4 million. Its expansion beyond CoreWeave could diversify sales from 2027, but the next projects require additional financing rather than the same customer funding that supported its existing conversions.
Meta is rolling out voice conversations and animated avatars for Muse in the US. The update adds personality to a cloud-based task agent whose costs, permissions and reliability matter more than its expressions.
Anthropic is rolling out Personal Finance to paid Claude users in the US, using Plaid to read bank data. Its pitch rests on ad-free answers and privacy controls, while its own documentation warns that balances and budgets can be incomplete.
OpenAI president Greg Brockman predicts significant medical advances within 12 to 18 months and raises the possibility of death becoming a choice. OpenAI’s cell research and a separate early drug trial show narrower progress, with financial ties, safety questions and clinical testing still central to judging the promise.
Citrini Research is recasting its AI investments around relative value as it warns that financing and construction constraints threaten returns. Its reported Oracle–Nebius trade faces a crucial counterargument: Nebius has substantial customer commitments, although delivery remains conditional.
Shield AI has agreed to acquire Echodyne for up to $5.3 billion, backed by a $2.5 billion financing package. The radar maker brings a profitable, defense-heavy business and commercial customers, with a promise to keep supplying outside system builders.
Gemini Omni 1.2 Flash lowers Google's quoted 720p generation price to $0.07 per second and extends native clips from 10 to 15 seconds. Editing limits and early commercial tests show why cheaper output does not guarantee the same savings on finished videos.
Mercedes-Benz plans to add Liquid AI’s on-device models to its North American voice assistant. The deal combines local processing with existing cloud AI, while customer availability, measured performance and contract economics remain unconfirmed.
Mistral is expanding Studio with managed agents, apps and workflow tools, extending its existing enterprise platform. Its private-deployment pitch sits alongside cloud-hosted app services and vendor-stored agent sessions, making the chosen configuration central to customer control.
Meta president Dina Powell McCormick expects tougher conditions for new US data centers after Democratic election victories, but says states still want investment. The test is whether electricity commitments, local jobs and tax benefits can secure community support.
OpenAI is releasing GPT-6 Cyber to selected verified defenders with stronger monitoring. Its Linux research shows the scale and cost of automated exploitation, while mixed benchmark results and unfinished patching limit what the launch proves about safer software.
Black Hills plans $1.8 billion of gas generation under agreements to serve Google’s proposed $17 billion Cheyenne campus through 2048. Google has advanced equipment funding, but delivery, fuel-cell conditions and local impacts remain unresolved.
Dropbox agreed to buy Tigris for about $350 million in cash and unveiled DBX Agent Cloud, starting with standalone object storage. The expansion accompanies AI actions for existing file customers, while pricing and evidence of competitive performance and productivity gains remain absent from the announcement.
ElevenMusic creators can now publish directly to SoundCloud, with a three-month Pro offer for Artist Pro subscribers. The integration shortens distribution while preserving disclosure and rights requirements.
A researcher demonstrated how a browser extension could use Claude Dispatch to execute code through a connected desktop agent. Anthropic reportedly acknowledged the flaw in May and proposed retiring Dispatch, but existing users still have access, leaving questions about authorization across devices.