What should a shopping agent do when nothing in the catalog fits what the customer needs? Visa’s answer, built into the benchmark it released on November 5, is the right one: don’t buy. That rule is why the headline number is so harsh. It is also why the more useful number sits one layer down, in what the agents did when they got it wrong.
The Visa Agentic Commerce Benchmark ran 22 models from 12 labs through 15 shopping scenarios: a TV, a laptop, a personal care appliance, three attempts each. Across all of it, the models passed 57.2% of their runs and failed 42.8%. The best passed 82.2%; the worst, 28.9%.
Most coverage stopped there. Anyone building or buying an agent that touches a payment method should keep reading the result set.
A test that rewards asking, not guessing
The design is what makes these numbers worth taking seriously. The simulated customer is itself an agent, and it volunteers some information but not all of it. In the TV scenario, Fortune lists requirements such as compatibility with a child’s gaming console, a 30-day return window, and Dolby Vision. The shopping agent has to draw those out in conversation.
According to The Next Web’s account of Visa’s release, an AI passes a run only if it finds every requirement set in the task and makes the right call to buy or not. Getting lucky does not count: the agent has to have gathered enough to justify its choice. Visa says the full research and interactive tests are public on its Payments Frontier site.
Two splits in the data point at what separates good agents from bad ones. The five models that uncovered the most hidden requirements averaged 70.2%, against 44.4% for the five that uncovered the fewest. And the five most expensive models per run passed 67.1%, while the five cheapest passed 50.2%.
Price buys about 17 points. Asking questions buys about 26. The cheapest improvement available to anyone building a commerce agent is not a bigger model; it is an agent that refuses to proceed until it has asked about budget, compatibility and return terms.
Visa did not name the models in its release. Fortune, working from the results site, reported that Google’s Gemini and Meta’s Muse agents tied at the top at 82%, a Claude agent came third at 76%, and an OpenAI agent finished sixth at 64%. Fortune did not give versions for the Claude or OpenAI entries, so treat any model-level comparison as provisional until you have read the leaderboard yourself.
Half the failures still ended at checkout
Here is the number that should change how these systems get built. In 54% of failed runs, the agent still made a purchase: either a product that did not meet the customer’s needs, or a purchase made before the agent had found all the requirements.
Run the arithmetic on Visa’s published figures (this is our calculation, not Visa’s): 22 models times 15 scenarios times 3 attempts is 990 runs. A 42.8% failure rate is about 424 failed runs. If 54% of those ended in a purchase, roughly 229 runs, or about 23% of everything Visa tested, ended with money spent on a decision that did not pass.
| Figure | Value | What it tells a builder |
|---|---|---|
| Overall pass rate | 57.2% | The field average is not deployable without guardrails |
| Best model | 82.2% | Even the leaders fail 17.8% of runs |
| Failed runs that still bought | 54% | The default failure mode is spending, not stopping |
| All three attempts passed | 34.2% | A single successful test proves little |
| Top-5 vs bottom-5 at finding requirements | 70.2% vs 44.4% | Elicitation is the skill that matters most |
Sources: Visa’s release as reported by The Next Web and Fintech News Singapore. The “what it tells a builder” column is this site’s reading.
The other 46% of failures are the ones you want. An agent that fails to find a suitable product, or declines when it should have bought, costs a lost sale. An agent that buys the wrong television costs a return, a refund, possibly a chargeback, and a customer who stops delegating. The benchmark shows that today’s agents lean toward the expensive failure.
One clean run tells you almost nothing
The consistency figure is the one that should worry anyone running a pilot. When a model attempted the same scenario three times, all three attempts passed in only 34.2% of model-scenario combinations. Visa also found, per The Next Web, that models often failed scenarios they had already passed once.
That breaks the most common way companies evaluate an agent: a handful of demo flows, each run once, watched by the team that built it. If the flow works, it ships. Visa’s data says a pass on Tuesday is weak evidence of a pass on Wednesday. Evaluation for anything that spends money needs repeated trials of the same scenario and a pass rate per scenario, not a single green check. We covered how to build that kind of harness in our guide to writing evals instead of trusting leaderboards.
Make the wrong answer cheap
Visa’s own framing is careful. Rubail Birwadker, its global head of growth, told Fortune: “We’re not there yet from a trust or technology perspective. But we’ll soon get into a world where consumers can trust and delegate.” Visa says it will extend the benchmark to checkout, payments and order placement. Fortune adds the part nobody has settled: it is still unclear who bears the liability when agents make rogue purchases.
Consumers have already said what they want in the meantime. In the 2026 FIS Consumer Pulse Survey of 2,000 U.S. adults, conducted with Kantar from September 16 to 30, 50% want to approve every transaction an AI assistant makes. Asked for their top three requirements, 58% picked a refund guarantee for mistakes and 56% picked access to a human who can review or override decisions. Visa’s own Trust Index found only 23% of U.S. consumers trust AI to make payments for them.
Put the benchmark and the survey side by side and the design brief writes itself. Until accuracy improves, the job is to move failures from “bought the wrong thing” to “stopped and asked”:
- Make no purchase the default outcome. The agent proposes; a person or a hard rule approves. Half of FIS respondents already expect this.
- Require the requirement list before checkout. Have the agent state budget, compatibility and return terms back to the user, and block payment until each one is confirmed. Visa’s 70.2% versus 44.4% split says this is where accuracy lives.
- Prefer merchants with structured attributes. An agent can verify “Dolby Vision” or “30-day returns” only if the product page exposes them as data. The access standards fight we covered in Amazon blocking shopping agents while Meta built a protocol to let them in is partly a fight over this.
- Cap the spend per task and per day with the card or wallet controls, not with instructions in the prompt.
- Log the conversation that led to the purchase. When a dispute arrives, the record of which requirements the agent found is the evidence. Our piece on agent observability and attestation covers what that log needs to hold.
None of this is new to anyone who has run purchasing in a large organization. Running operations at a large telecom, the purchasing controls I trusted most were never the ones that made a buyer smarter. They were the ones that made a wrong order cheap to unwind: an approval threshold, a receiving step before payment released, return terms negotiated before the purchase order went out. Agents need the same treatment, because Visa has now measured what happens without it.
The liability question will not wait for the technology either. Platforms are already writing AI risk onto their customers, as Meta’s new commercial terms show, and insurers have spent 2026 excluding agent errors from standard policies. A company that lets an agent spend on a customer’s behalf, with a 54% chance that any given failure ends in a charge, should assume the refund is coming out of its own margin.
Visa wants people to try the test before Black Friday. Builders should do more than try it: run their own agent through the public scenarios several times each, count how often a failure ends in a purchase, and treat that number, not the pass rate, as the one that decides whether the agent gets a card.