Google, OpenAI, Anthropic, and Microsoft Back New Robots.txt Signal to Block AI Scraping

Original headline: Google Joins OpenAI And Anthropic In Promising New Scraping Controls

GAI Press Editors · Sunday, November 1, 2026 · AI summary

Google, OpenAI, Anthropic and Microsoft have pledged to support a new common robots.txt signal that allows websites to opt out of AI training data scraping. The signal lets site owners block crawlers from harvesting content for large language model training, aiming to improve copyright and privacy safeguards.

The move offers a transparent way to curb unauthorized data harvesting, strengthening copyright and privacy protections for content creators.

Research angleHow does an opt‑out mechanism for AI data scraping affect firms' willingness to share data? — Technology Acceptance Model (TAM)

Read the original at Engadget ↗Oct 31, 13:34 KST

3 related reports

Google, Anthropic and others to support new AI crawling opt-out standard — SiliconANGLE
OpenAI plans to train models on Daybreak Blue cybersecurity session data that was previously off-limits — The Decoder
Google Deepmind creates a global map of forests with satellite data and AI — The Decoder

This story could become your next paper

HJITM works with researchers from study design through surveys, analysis and submission to international journals in generative AI and technology management. Bring us your research question.

Ask about research design →Free statistics tool ↗About HJITM

← 1. Anthropic’s Watermark Expansion Leaves B…3. Google Launches Gemini 4 Carbon, AI Mode… →

This item is an AI-generated summary of reporting by Engadget. Please check the original article for accuracy. Copyright in the original article belongs to Engadget.