Two Economies of AI Search: Discovery vs. Consumption

When a shopper asks an AI assistant “what’s the best digital planner for ADHD?” your listing might be the source material for the answer they read. But they will probably never click through to your page.

A 2025 Pew Research Center study found that when Google shows an AI summary, users clicked on a traditional search result link just 8% of the time — about half the 15% click rate on pages without a summary. They clicked a link inside the AI summary itself only 1% of the time. And 26% ended their browsing session entirely after seeing an AI-generated answer, compared with 16% on traditional results pages.

For marketplace sellers, that data describes a shift that is already reshaping how buyers find products. Your listing content — titles, descriptions, attributes, images — now powers AI answers that shoppers read directly, often without visiting your store. The question is no longer just “am I being crawled?” but “what happens to my content after it’s read?”

The answer, according to Cloudflare’s July 2026 Content Independence Day report, is that the economics of search are splitting into two distinct models. Understanding which one your listings live in determines where you should spend your optimization effort.

The crawl data tells a clear story

Cloudflare, whose network sits in front of more than 20% of all web domains, publishes the most granular crawl data available. Their June 2026 report reveals how crawler behavior has shifted in a single year:

• 52% of crawler requests are now for AI training, up from 22% in Spring 2025. Pure search crawling — the kind that funnels shoppers back — represents a declining minority of crawler activity.
• Mixed-use crawlers (those combining search, agent use, and training) represent over 36% of activity. These are the bots that make it impossible to distinguish whether your content is being indexed for discovery or absorbed for model improvement.
• More than 50% of all traffic on the internet is now non-human. Agent traffic crossed this threshold for the first time in 2026.

The crawl-to-referral ratio is where the asymmetry becomes concrete. Cloudflare’s Attribution Business Insights dashboard, launched July 1, 2026, measures how many times a bot crawls your content for every visitor it sends back. Traditional search engines operated at a relatively balanced ratio — a few crawls per referral. AI crawlers operate at ratios ranging from 118:1 to nearly 50,000:1.

In other words, an AI system might read your premium listing content tens of thousands of times to send back a single click. The rest of those reads powered answers that shoppers consumed without ever visiting your page.

Two economies of AI search

The old model of search was simple: let crawlers in, get indexed, receive visitors, convert visitors into sales. Being crawled and getting visited were the same transaction.

That model is splitting. Cloudflare’s report describes the divergence clearly: “Search drives users to content, while AI-powered experiences increasingly summarize and reuse it without requiring users to visit the source.” Discovery and consumption now serve different purposes, and they reward different seller behaviors.

The discovery economy is the familiar one. A search crawler reads your listing, indexes it, and when a shopper’s query matches, your product appears as a link the shopper can click. You optimize for this by being crawlable, having strong titles, and ranking well. This economy still sends real visitors, even if fewer than before.

The consumption economy is the new one. An answer engine reads your listing, extracts the relevant facts, and synthesizes them into an answer the shopper reads directly. The shopper may never see your link, let alone click it. Your content has been consumed — its facts, attributes, and comparisons folded into someone else’s answer — but the visit never happens. The Pew data quantifies how often that consumption occurs without a corresponding click: 92% of the time when an AI summary is present.

Marketplace sellers live in both economies simultaneously. A shopper comparing Bluetooth headphones on Google AI Mode reads a synthesized answer that may pull from your product page, a Reddit thread, and a YouTube review — all without clicking any of them. Your listing’s content has been consumed, but no visit was generated. Meanwhile, the same listing might appear as a clickable link in a separate discovery result that a different shopper does click.

The Google convergence problem

Not all search platforms treat these two economies equally. Cloudflare’s report highlights a structural issue that directly affects marketplace sellers who depend on Google for visibility.

Most leading AI companies — OpenAI, Anthropic, Perplexity — operate separate crawlers for different purposes. OpenAI, for example, runs OAI-SearchBot for search indexing, GPTBot for training, and ChatGPT-User for agent actions. This separation lets site owners allow discovery while blocking training, preserving the discovery economy’s referral pipeline.

Google does not separate its crawlers. According to Cloudflare, Google’s mixed-use bot gives it access to approximately 2x more information than leading AI companies because website owners cannot participate in Google’s search ecosystem without also participating in Google’s AI ecosystem. The same crawl that indexes your listing for traditional search also feeds Google’s AI Overviews and AI Mode — the surfaces where the consumption economy dominates and click-through rates are lowest.

For marketplace sellers on platforms like Etsy, Amazon, or Shopify, this is largely out of your hands. The platform controls crawler access, not the individual seller. What you control is the quality and completeness of the content those crawlers find when they arrive.

The licensing economy is emerging

The consumption economy’s problem — content gets used without compensation — is starting to find a market solution. Cloudflare’s July 2026 report identifies more than 50 publisher-AI licensing agreements signed since 2023, and the company is building infrastructure to scale that model.

The evolution runs from “Pay Per Crawl” to “Pay Per Use.” Cloudflare launched Pay Per Crawl in 2025 so publishers could charge AI companies for crawling their content. But crawling is a crude measure: a page might be fetched once and cited in thousands of answers, or fetched repeatedly and never used at all. The newer model ties payment to actual value — when your content appears in an AI answer, you get compensated.

Two partners illustrate the approach. Ceramic.ai has built a “pay-per-query” model where publishers who opt in receive payment when their content appears in search results. You.com lets agents pay on demand for specific premium content. Cloudflare provides the infrastructure layer; the payment models come from the AI companies themselves.

For large publishers with original journalism and premium content, licensing revenue is already material. For marketplace sellers, the licensing economy is still emerging and primarily relevant to those who run their own storefronts with original, differentiated content — custom photography, unique written guides, proprietary comparison data. If your listing is commodity attributes (product title, price, standard specs), the licensing opportunity is limited. If your listing contains original analysis or creative work that AI systems want to cite, it may eventually be worth something.

The more immediate implication for sellers is strategic, not transactional: the market is moving from “block everything” to “value everything.” The sellers who understand which of their content assets are discoverable, which are consumable, and which are worth licensing will be better positioned as the infrastructure matures.

What marketplace sellers should do now

The two-economy framework changes where sellers should focus their effort.

Optimize for citation, not just clicks. In the discovery economy, you optimize to earn the click. In the consumption economy, you optimize to be the source the AI answer is built from. That means your listing content should be structured so that individual paragraphs, data points, and comparisons can stand alone as self-contained answers. Each section of your listing — product description, specifications, usage instructions, comparison details — should directly answer a question a shopper might ask.

Fill every attribute an AI agent checks. An anonymized FirstShelf audit of 22 listings over 90 days found an average AI-readiness score of 50.1 out of 100. The most common missing entities were microchip status, software compatibility, refund and returns rules, authenticity guarantees, and file or page counts. These are exactly the attributes an answer engine extracts when synthesizing a product recommendation — and exactly what determines whether your listing makes it into the answer or gets skipped for a more complete competitor.

Track AI visibility separately from traffic. Google Search Console does not separate AI Mode data from regular search. The Pew study shows that 18% of Google searches in March 2025 produced an AI summary, and that share is growing. If you only measure visits, you are blind to how often your content appears in answers that never generate a click. Use specialized tools or manual testing to check whether your products appear in AI answers for your category’s key queries.

If you run your own storefront, separate your access controls. The discovery economy rewards allowing search crawlers. The consumption and training economies may not. Cloudflare’s September 15, 2026 default will block Training and Agent bots on ad-monetized pages for new domains while keeping Search allowed. If you have your own site on Cloudflare, confirm your settings explicitly rather than inheriting defaults — and allow search crawlers like OAI-SearchBot while evaluating whether to restrict training crawlers like GPTBot.

Marketplace sellers: focus on what you control. On Etsy, Amazon, and similar platforms, the marketplace controls crawler access. Your lever is listing completeness — the structured attributes and content quality that determine whether you are cited inside an AI answer or passed over. The platform gets you crawled; you determine whether you are usable once the crawler arrives.

How FirstShelf can help

The two-economy shift means that listing completeness is no longer just an SEO concern — it determines whether your content is usable in the consumption economy where clicks don’t happen but citations do. FirstShelf audits your listings against the same machine-readable signals an answer engine reads: semantic density, structure quality, entity authority, and platform compliance. It identifies the exact attributes to fill — the ones an AI agent checks before including your product in a synthesized answer.

An anonymized audit of 22 listings across 90 days found an average score of 50.1 out of 100, with the biggest gaps in entity authority (31.8 average) and structure quality (33.7 average). Those are the dimensions that determine whether your listing content can be extracted, understood, and cited — or whether it gets skipped for a competitor whose attributes are complete.