The Crawler Split Is a Listing Job

If you sell on Shopify, Etsy, or Amazon, you probably do not have a “Block AI bots” switch. You have a product. Search, Agent, and Training are still three different jobs — they just live on the listing, not in robots.txt.

Cloudflare named the split Search, Agent, and Training. OpenAI documents the same three jobs as separate user agents: OAI-SearchBot for search, ChatGPT-User for user-triggered fetches, and GPTBot for training. Each setting is independent. A site can allow search and disallow training. OpenAI is explicit that sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers,” though they can still appear as navigational links.

That is webmaster control. Most marketplace sellers never get it. Etsy and Amazon own the crawler file. On Shopify, Catalog and Agentic surfaces read the product record Shopify already has. The operating method for GEO for marketplace sellers is the Admin product, not a CDN policy.

If you do run your own domain on Cloudflare, the file-level Disallow vs Block choice is covered in Cloudflare Now Writes Your robots.txt. This article is the other half: what to inspect on the listing when you cannot (or should not) treat crawler access as the work.

Search: can an answer system describe this product?

Search is the “be understood later” job. Shopify’s own Catalog guidance lists the product details AI platforms and shopping sites consider: Title, Description, Images, Type, Vendor, Collections, Tags, Barcode (ISBN, UPC, GTIN), Variants including Option name, and — on the Agentic plan — External product URL.

Those are Admin fields.

• Title is the name customers see. Google Merchant Center requires a title that clearly identifies the product (1–150 characters) and tells you to distinguish variants in the title, without stuffing price or promo language into it.
• Description is the rich-text body on the product. Shopify tells resellers not to paste the manufacturer’s exact description, because you want the listing unique to search engines. Google requires a description of the product itself (up to 5,000 characters), not store history, not other products, not promotional shipping claims.
• Category unlocks category metafields from Shopify’s Standard Product Taxonomy — size, fabric, neckline, and similar attributes that marketplaces and search engines already know how to read.
• Barcode is the GTIN/UPC/EAN/ISBN. Shopify notes some channels require it before they will publish the product. Google treats a correct GTIN as the way it matches the same product across the web; a wrong one can get the offer disapproved.

If those fields are empty, contradictory, or copied from a manufacturer page every other seller uses, blocking or allowing a crawler does not create a distinct product for an answer engine to cite. Shopify Product GEO is the field-level version of that search job.

Agent: can something acting for a shopper commit?

Agent is the “act now” job. OpenAI’s ChatGPT-User agent fetches a page because a person asked. OpenAI says those actions are user-initiated, so robots.txt rules may not apply, and ChatGPT-User is not what decides Search placement.

So even a store that “blocks AI bots” can still be hit when a shopper’s assistant tries to buy. The question is whether the listing has the facts an agent needs to finish.

Shopify’s Catalog optimization page is blunt about policies: keep return, shipping, and related store policies complete and up to date so that when a customer asks about returns in an AI interaction, the agent can reference what you stored in Shopify — Settings → Policies, not a paragraph buried in a blog post.

On the product itself, the Agent job is:

• Variants and option names — size, color, pack size. Shopify Catalog explicitly includes option names in the AI/shopping field list. An agent that cannot tell which variant is in stock cannot buy.
• Inventory / SKU — whether that variant is actually available.
• Shipping on the product (physical vs not, origin) plus the store shipping policy.
• Price that matches the live offer, without stuffing sale language into Title or Description (Google’s merchant rules put promo facts in price/shipping attributes, not in the prose).

Shopify also points merchants at the Knowledge Base app for FAQs that AI platforms use. That is a store-level surface, not a product field; we covered how those FAQs get compiled in AI Agents Quote FAQs You Never Wrote. Do not treat KB copy as a substitute for variant and policy fields on the product.

Training: you cannot un-train a listing field

Training is the “absorb this into a model” job. GPTBot is OpenAI’s training crawler. Disallowing it is a robots.txt preference, not a Shopify Admin toggle.

On a marketplace, you usually cannot set that preference. On your own Shopify domain you might — that is the Aug 24 article. Either way, a listing field does not retract content already used for training.

What you can control on the product is how interchangeable the listing is:

• Write a description that is not the manufacturer’s boilerplate. Shopify already asks resellers to do this for search uniqueness; it is also the only listing-level way to stop handing every model the same paragraph ten other shops published.
• Use your own product photos, 3D, and video in Media. Stock-pack shots and manufacturer renders are the easiest assets to reuse without citing you.
• Fill category metafields with the actual garment/product facts (fabric, sleeve length, compatibility) instead of leaving them blank so a model has to invent them.

That is not legal advice and not a guarantee that a model will honor your preference. It is the listing work that remains after you accept you do not own the crawler file.

What to inspect on one product this week

Open one live product in Shopify Admin. For each job, check the field — not a dashboard score.

Search (cite): Title names the product and the distinguishing variant fact. Description describes this product in your words, in the first screen, without shipping slogans. Category is set so metafields appear. Barcode is the real GTIN if the product has one.

Agent (buy): Every option name is a real shopper word (Size, Color), not “Option1.” Inventory is true. Shipping policy and return policy exist under Settings → Policies and match what you would tell a customer in chat.

Training (reuse): Description is not a paste of the brand’s Amazon copy. Media includes at least one original photo of the actual item.

If you also have a custom domain on Cloudflare, set Search allowed and decide Training as Disallow vs Block there — then come back to this list. Access without these fields is a page an agent can fetch and still not use.

How FirstShelf can help

FirstShelf is built as GEO for marketplace sellers: make the product visible and recommendable by AI by inspecting the listing fields those systems actually read. A listing audit walks Title, Description, category metafields, variants, media, and policy completeness — the Search and Agent jobs above — rather than generating a robots.txt you may not control.