Cloudflare Now Writes Your robots.txt

For most of the web’s history, robots.txt was a file you wrote by hand, updated rarely, and hoped crawlers respected. On August 21, 2026, Cloudflare announced Bot Preference Sync, and for any storefront behind its network that file now increasingly writes itself: the preferences you set in a dashboard get published straight into your robots.txt, kept in sync automatically, on every plan from Free to Enterprise.

The detail marketplace sellers should not miss is smaller than the announcement but more consequential: the Training control now has two distinct outcomes. Block denies a crawler’s requests at the edge. Disallow writes a “no training” preference into robots.txt that cooperating crawlers honor while continuing to index your pages for search. One setting removes you from an engine’s index. The other keeps you in it. Choosing between them, knowingly, is the new job.

Here is what shipped, how the two options actually differ, what a generated robots.txt means under the protocol rules that govern it, and what this changes for sellers on their own domain versus sellers inside Etsy, Amazon, or Shopify.

What Bot Preference Sync actually does

Bot Preference Sync connects two layers that used to drift apart: what you configure and what you publish. Cloudflare’s own framing for the feature is the mismatch problem — cases where “your robots.txt states that a crawler is Disallowed from accessing your website, while your enforcement rules actually don’t block that crawler.” When your stated preferences and your enforced rules disagree, Cloudflare notes, some crawlers treat it as a basis to disregard your preferences entirely.

The sync closes that gap mechanically. When enabled, it generates or updates your robots.txt to reflect the AI bot configuration you have set for Search, Agent, and Training traffic — the three-way taxonomy Cloudflare introduced on July 1, 2026. If you already have a robots.txt file, the generated content is prepended between marked sections, so your existing directives survive. The bot lists themselves come from BotBase, Cloudflare’s directory of tracked bots, and are updated periodically as classifications change.

Availability is broad by design: all customers, Free tier through Enterprise, with the feature on by default for new customers. Existing customers using the older managed robots.txt feature will be prompted to review and confirm their preferences during the transition. {{stat:1}}

For a seller running their own storefront, that means the file crawlers read is no longer a static artifact you own exclusively. It is the published reflection of policy decisions made in a control panel — and those decisions just became more consequential, because one of them changed meaning.

Disallow versus Block: the choice that decides your search visibility

When Cloudflare split the single “AI bots” toggle into Search, Agent, and Training in July 2026, the Training control was effectively a block. The August 21 announcement refines it into two options with different consequences:

Block is enforcement. The crawler’s requests are denied at the edge of the network. Whatever the crawler wanted — indexing, training, acting on a shopper’s behalf — it does not get through.

Disallow is a published preference. A “no training” directive is written to your robots.txt, and mixed-use crawlers that meet Cloudflare’s transparency requirements can still access your content for search indexing, because they give you a way to verify how your data is used. Your search visibility with cooperating crawlers is unaffected.

Why this middle option works is not Cloudflare’s assertion alone. OpenAI’s crawler documentation independently documents the same separation on its side of the exchange: OAI-SearchBot surfaces sites in ChatGPT’s search results, GPTBot crawls for model training, and each setting is independent — “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot.” OpenAI also states the cost of getting it backwards: sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers,” though they can still appear as navigational links. {{stat:2}}

That is the practical stake in the Block-versus-Disallow choice. Blocking a mixed-use crawler to protect your content from training can take your products out of the answers where shoppers now start. Disallowing training — for crawlers that honor the distinction — protects the content while keeping the citation path open.

The transparency test behind the middle option

Disallow is not unconditional. Cloudflare frames it as making transparency “the price of admission”: bot operators that mix search and training behind one user agent must earn the right to keep crawling for search after a site says no to training. The announcement lists four requirements — the bot must respect a no-training preference through some mechanism, give site owners a way to opt out of AI summaries, provide URL-level visibility into which pages were used for training along with search metrics, and be able to show publicly that disallowing training does not hurt traditional search results.

Operators that meet the criteria are tracked in a public AI bot transparency section of Cloudflare Radar, including examples of best practices honored and violated. Crawlers that do not provide transparency “don’t get the benefit of the doubt” — they are still blocked when you disallow training.

For sellers, the significance is structural rather than operational: the industry is converging on verifiable, per-use crawler separation instead of a single “AI bot” identity. The more operators that separate their crawlers and disclose their behavior, the more the Disallow option does what it promises. Until then, the guarantee only covers crawlers inside the transparency framework.

What a generated robots.txt means under protocol rules

A robots.txt written by your CDN is still a robots.txt, governed by the same rules as one you typed yourself. Three of those rules matter for sellers making Block-versus-Disallow decisions:

It is advisory, not enforcement. RFC 9309, the IETF standard for the Robots Exclusion Protocol, is blunt on this point: the rules “are not a form of access authorization.” Robots.txt works because crawlers choose to honor it. That is precisely why the sync exists — Cloudflare pairs the published preference with edge enforcement for the categories you block, so cooperating crawlers read the file and non-cooperating ones hit the rule. Google’s own documentation echoes the boundary from the operator side: robots.txt “is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google” — for that, Google directs site owners to noindex or password protection.

Precedence is deterministic. The standard requires crawlers to use the most specific matching rule, and when an allow and a disallow rule match equally, the allow rule wins. Because sync content is prepended to your existing file, your own more specific rules can still override generated ones — worth knowing if you maintain custom exceptions for a particular partner crawler.

Changes propagate on a delay, and the file has a size ceiling. RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours, and OpenAI states roughly the same delay for its systems. Google enforces a 500 kibibyte file size limit and ignores content beyond it — a real constraint for large catalogs whose generated section accumulates user-agent lines as bot classifications update. Google’s own advice for oversized files is consolidation, such as grouping excluded material under shared directory rules. {{stat:3}}

Who controls the file: the marketplace split

Everything above applies to sellers who run a storefront on their own domain. If you sell on Etsy or Amazon, the platform controls robots.txt and crawler policy, not you — the practical lever remains what it was before this announcement: the completeness of your listing’s structured attributes, which decides whether you are cited or skipped once a crawler does arrive. That is the same split we drew in our breakdown of the three-way crawler controls, and it has not moved.

Shopify sellers sit in between: the storefront runs on your domain, Shopify manages much of the technical surface, and crawler-facing decisions increasingly arrive as platform defaults rather than hand-edited files. The skill that compounds across all three cases is GEO for marketplace sellers — making listings legible to the engines that read them — because access decisions only matter when there is something coherent behind the door. The enforcement side of that equation, and the risk of defenses colliding with shopping agents you actually want, is a separate problem we covered in our analysis of bot defenses meeting shopping agents.

What to do before the rollout completes

Bot Preference Sync reaches all customers “in the coming week” per the announcement, and the September 15, 2026 default changes Cloudflare set out in July are close behind. FirstShelf’s recommended sequence for sellers on their own domain:

1. Audit your current robots.txt now. Note which AI crawlers you currently allow or disallow, so you can recognize what changes when generated content appears between the sync markers.
2. Decide Training as Disallow, not Block, unless you have a reason not to. This is a FirstShelf recommendation, not a Cloudflare instruction: for sellers whose priority is visibility, Disallow preserves search access with cooperating crawlers while stating the no-training preference. Sellers whose content is highly original — patterns, photography, written guides — may still prefer the harder line of Block and accept the search cost.
3. Keep Search allowed. The search crawler is the one that funnels shoppers back. Opting out of a search bot removes you from that engine’s answers, per the operators’ own documentation.
4. Decide the Agent door deliberately. Agent traffic is the category that completes purchases on a shopper’s behalf. Closing it costs you nothing in citations, but it removes you from agent-mediated checkout — a trade-off to choose, not inherit.
5. Watch file size if you run a large catalog. If your robots.txt grows past 500 KiB with generated user-agent lines, Google ignores the overflow. Consolidate directory-level rules before that happens.
6. Expect roughly a 24-hour lag after any change. Both the RFC’s caching guidance and OpenAI’s stated timing mean same-day testing of a robots.txt change will mislead you.

Marketplace sellers can skip the file work entirely and spend the same hour on listing attributes — brand, materials, compatibility, shipping, and returns — which is the surface they actually control.

How FirstShelf can help

FirstShelf audits your listings against the same machine-readable signals a search crawler or shopping agent reads — structure quality, entity authority, semantic density, and platform compliance — and shows you the exact attributes to fill before a bot ever reaches them. Crawl preferences decide whether the door is open; FirstShelf makes sure there is something coherent behind it. Run the audit before the rollout reaches your storefront, so the preference you publish points at listings that can actually be cited.