By Tech & SEO News Desk
Published: September 2024

As generative artificial intelligence continues to reshape the digital landscape, website publishers are increasingly seeking ways to protect their intellectual property. A central point of confusion for webmasters has been whether standard robots.txt files can be configured to permit AI crawlers to index content for search visibility while simultaneously barring that same data from being utilized in large language model (LLM) training.

According to recent clarifications from Google Search Relations experts Gary Illyes and Cherry Prommawin, the short answer is no. Speaking on the second episode of Google Search Central’s video series, The Non-Canonical Cabinet, Illyes broke down the technical boundaries of the internet standard governing web crawling, emphasizing that robots.txt dictates access, not post-crawl data usage.

The discussion highlights a critical structural gap in modern web administration: while publishers want granular control over how their text, images, and code feed artificial intelligence, the existing technical infrastructure separates crawling permissions from usage rights.

Main Facts: What robots.txt Can and Cannot Do

At its core, robots.txt is an access-control protocol. As formally defined by Internet Engineering Task Force (IETF) standard RFC 9309, the file governs which URLs a web crawler is permitted to request from a server. It does not dictate what an automated system does with the collected data after it has already harvested it.

During the episode, Illyes addressed the common misconception that publishers can insert custom, improvised commands into a robots.txt file—such as a fictional No-AI-Chicken-Training directive—to block generative AI training while leaving standard search indexing intact. Such custom directives are entirely ignored by parsers.

Under Google’s official robots.txt specification, only four elements are natively supported:

• User-Agent: Identifies the specific crawler the rule applies to.
• Allow: Permits crawling of specific URL paths.
• Disallow: Blocks crawling of specific URL paths.
• Sitemap: Points crawlers to XML sitemaps.

Furthermore, Section 2.2.4 of RFC 9309 explicitly grants crawlers the autonomy to decide how they process fetched records outside the scope of the protocol. Consequently, if a webmaster wishes to keep a specific AI crawler off their entire website, they must explicitly target that crawler’s specific user-agent token within the robots.txt file using standard Disallow commands.

Chronology of Events and Policy Evolution

The debate over AI crawling, search visibility, and content rights has accelerated rapidly over the past two years, marked by several key milestones:

• June 2022 (RFC 9309 Published): The IETF formally establishes RFC 9309 as the internet standard for the Robots Exclusion Protocol, defining the baseline rules for web crawlers.
• August 2023 (Introduction of Google-Extended): Google introduces the Google-Extended token for robots.txt, giving site owners a mechanism to manage whether their content can be used to train Gemini models and power AI features without affecting traditional Google Search rankings.
• August 2024 (IETF Draft Proposal): Gary Illyes, alongside Mozilla’s Martin Thomson, co-authors an IETF Internet-Draft proposing an official Content-Usage rule for robots.txt and a matching HTTP header to standardize usage preferences.
• September 2024 (Vocabulary Draft Updates): A separate vocabulary draft introduces labels such as train-ai=n, though the IETF notes this currently reflects no formal working group consensus.
• Wednesday (Google Search Central Episode): Google airs the second episode of The Non-Canonical Cabinet, featuring Illyes and Cherry Prommawin addressing edge-case technical SEO questions, including AI crawling limitations and percent-encoding for non-English paths.
• February 2027 (Projected Expiration): The current IETF Content-Usage draft is officially set to expire, placing a soft timeline on the standardization efforts unless renewed or advanced.

Supporting Data and Technical Nuances

To navigate the complexities of modern web crawling safely, webmasters must understand how different tools interact with parsers, token architectures, and character encoding.

The Google-Extended Mechanism

Unlike traditional user-agents that fetch specific pages under a named bot, Google-Extended is a token with no independent user-agent of its own. Google crawls pages using its standard web crawlers, but checks the Google-Extended rule in the site’s robots.txt to determine if the retrieved data may be used for:

1. Training future Gemini models.
2. Grounding in Gemini Apps.
3. Grounding with Google Search on Vertex AI.

Google’s documentation explicitly states that utilizing the Google-Extended token does not affect a website’s inclusion, indexing, or ranking performance within standard Google Search. However, because this mechanism relies on Google’s proprietary product documentation rather than RFC 9309, it represents a platform-specific solution rather than a universal standard.

The IETF Content-Usage Draft

To bridge the gap between crawling and usage, the IETF draft co-authored by Illyes proposes adding a Content-Usage rule to robots.txt. If ultimately approved and adopted, the protocol would function as follows:

• Allow and Disallow would continue to manage foundational web crawling permissions.
• Content-Usage would specify post-fetch permissions exclusively for paths where the crawler already has explicit permission to fetch content.
• Proposed rule examples include parameters like train-ai=n to signal that allowed content should be withheld from machine learning pipelines.

Despite these efforts, progress is slow. The draft remains a Working Group Document and has not yet been submitted to the IETF’s Steering Group (IESG) for formal review. The working group’s projected timeline targets submission around August 2026, with the document scheduled to expire in February 2027 if not superseded.

Percent-Encoding Non-English Paths

During the video broadcast, Cherry Prommawin offered crucial technical advice regarding path formatting. For URLs containing non-English characters, localized scripts, or emojis, webmasters should consistently employ percent-encoded forms within their robots.txt rules.

While Google’s advanced parser natively handles raw UTF-8 characters by encoding them internally, older or less sophisticated web crawlers across the broader internet frequently stumble when processing raw Unicode or special characters. Utilizing percent-encoding ensures cross-bot compatibility, preventing unintended indexing loopholes caused by parser errors.

Official Responses and Industry Perspectives

The guidance provided by Google Search Relations highlights a growing tension between major technology platforms and content creators. Publishers argue that forcing an "all-or-nothing" choice—where allowing a bot to read a page inherently grants permission to ingest it into an LLM—violates the spirit of copyright and fair compensation.

Google maintains that its current architecture balances the needs of web discovery with publisher preferences via tokens like Google-Extended. By separating search crawling from training controls within its own ecosystem, Google provides a functional workaround while acknowledging the absence of a universal internet standard.

Standardization bodies like the IETF are working to address this vacuum through the Content-Usage draft. However, because the proposal is still in its drafting phase and lacks broad industry-wide consensus, webmasters have no immediate, universal mechanism to police AI model training without completely sacrificing organic search traffic.

Implications for Webmasters and SEO Professionals

The technical realities outlined by Google carry profound implications for website owners, digital marketers, and technical SEO specialists:

1. Audit Your robots.txt Files: Webmasters must review their existing configurations to ensure they are not relying on obsolete, non-standard custom strings (like imaginary AI-blocking rules) that parsers automatically discard.
2. Granular Bot Management: To block specific AI scrapers while maintaining visibility on traditional search engines, administrators must explicitly target the user-agent tokens of those individual AI scrapers.
3. Monitor Platform-Specific Tokens: Sites heavily reliant on Google traffic should evaluate whether to implement Google-Extended depending on their institutional stance on generative AI training.
4. Adopt Safe Character Formatting: Technical teams managing multilingual sites or URLs featuring special characters must adopt percent-encoding as a best practice to ensure universal crawler compliance.
5. Stay Informed on Standards: Because the IETF Content-Usage draft is evolving slowly, SEO professionals must monitor regulatory and protocol updates to adapt their compliance strategies as new internet standards take shape.

As the digital ecosystem awaits a unified, cross-industry protocol for AI content governance, webmasters must remain vigilant, utilizing the precise tools currently available to safeguard their digital assets without inadvertently crippling their search visibility.