AI Shopping Metrics Considered

AI-assisted shopping journeys on an ecommerce site may force merchants to reconsider what to measure, and when.

“Many familiar ecommerce metrics are built around the assumption that shoppers discover products by navigating between pages,” wrote Zhengyang Yang, a product designer at TikTok who works on AI-assisted shopping experiences, in an email exchange with Practical Ecommerce.

“A shopper who performed five searches and visited 10 product pages might now describe their requirements once, compare three recommendations, and proceed directly to a purchase,” wrote Yang.

From the shopper’s perspective, less “engagement” is progress, but in an analytics dashboard, it might look like decline.

Short sessions can signal satisfied shoppers.

Activity

Put another way, an AI-assisted session could generate lower search volume, fewer product-page visits, and shorter sessions even when the shopper is having a better experience.

Yet ecommerce marketing teams have often treated activity as a proxy for interest.

• More product views can suggest consideration.
• More search activity can indicate engagement with a catalog.
• Longer session duration may imply that a shopper is spending time evaluating products.

But the same numbers can mean something else.

• Five queries could mean the search experience is poor.
• Ten product-page visits could indicate incomplete product information.
• A long session could represent confusion.

AI-assisted shopping does not create these ambiguities, but it may expose them.

Feature Use

The same problem applies to metrics for on-site AI assistants.

When a merchant launches an embedded AI assistant, the first metrics usually measure feature use rather than the shopping experience. Typical examples are AI conversations, messages per session, time spent interacting, and repeat usage. Those metrics show whether shoppers use the AI assistant. They do not show that it works.

“A shopper exchanging 10 messages with an assistant might indicate strong engagement, but it could also mean the assistant repeatedly misunderstood her needs,” Yang wrote.

This idea that less “engagement” or interaction is better may seem counterintuitive. Content sites and social platforms typically seek more time and more interactions, but an embedded shopping assistant often aims for the opposite.

A shopper who explains a need once, receives two recommendations, compares them, and completes the purchase may have had a better experience than someone who spends 15 minutes chatting.

Most small and midsize ecommerce merchants do not build their own AI assistants; they install an external app. Many of those apps report the conversations, messages, and time spent, but not whether the experience was productive.

Certainly feature use matters. If shoppers ignore an assistant entirely, the merchant has a placement, usability, or awareness problem. But use shouldn’t be the only measure of an AI assistant’s success.

Recommended

The Race to Own AI Shopping

September 6, 2026

Decision Quality

A better way to measure an on-site AI assistant, according to Yang, is to track decision quality.

• Does the AI tool help shoppers identify relevant products efficiently?
• Does the assistant compare products accurately?
• Do product recommendations turn into conversions?

Those outcomes are difficult to summarize in a single dashboard number, but merchants can still measure the behavior.

Repeated searches may suggest that the first results missed the shopper’s intent. Repeatedly reformulating a request could show the same thing. Moving back and forth among the same products may indicate unresolved comparison questions.

Controlled experiments can help too. Yang recommended comparing AI-assisted and conventional shopping experiences for similar purchase intentions. The comparison could include product discovery, task completion, conversion, and other outcomes.

The proper metric to track may also depend on the shopping task. A customer replenishing a familiar household item may value speed, while someone buying an expensive appliance or technical product may need more comparison and explanation.

“Successful AI assistance should be measured by the quality and efficiency of shopping decisions, not simply how much time users spend interacting with AI,” Yang wrote.

Outcomes

Decision quality can extend beyond the initial conversion, too.

Yang recommended examining product returns, customer satisfaction surveys, and repeat purchasing behavior when evaluating an AI shopping assistant. The theory is that an assistant that helps shoppers buy faster may still perform poorly if those purchases lead to relatively more returns or disappointed customers.

For example, a shopper might quickly accept an AI recommendation. But if she then returns the item because it doesn’t fit, is incompatible, or doesn’t meet expectations, the shorter buyer’s journey didn’t improve the outcome.

In other words, marketers may need to measure the AI-assisted shopping journey over a longer period, well into the post-purchase phase.