Explainer
Where AI training data comes from: crawling, licensing and opt-outs
Much of the text used to train large AI models is scraped from the public web by automated crawlers, often via the free Common Crawl archive, and then filtered and cleaned before training.[1][2] Developers also license content, use data from their own customers or users, and generate synthetic data.[3][4] Site owners can ask crawlers to stay away with robots.txt, but the standard is a request rather than access control, and whether training on copyrighted works is lawful is being fought out in courts and regulation.[5][6]
This page explains how training data is gathered and how opt-outs work. It describes the legal debate but is not legal advice.
Why models need so much data
Developers often describe training a large language model in two phases.[7] OpenAI, as quoted by the US Copyright Office, describes pre-training as the step in which a massive amount of computing power and data is spent to teach a model the broad foundations of language, followed by post-training on smaller, carefully curated datasets.[7] A 2020 study found that language-model loss falls as a power law as model size, dataset size and training compute grow.[8] For the basics of training, see how machine learning works.
Web crawling and Common Crawl
The Copyright Office’s May 2025 report on AI training, released as a pre-publication version, says training data is acquired in many ways, and that a common practice is downloading publicly available data from the internet using automated scraping tools.[1] One particularly common source is text scraped by web crawlers, often obtained from Common Crawl.[1]
Common Crawl’s corpus contains petabytes of raw web pages, metadata and text extracts, collected regularly since 2008; the copy hosted on Amazon Web Services is free to access.[9] Scraped material can dominate a model’s diet. The Copyright Office notes that GPT-3 was trained on several scraped datasets, including Common Crawl and WebText, which made up 82% of the weight of its training mix.[10]
Filtering and curation
Raw web text is messy. The Copyright Office notes that scraped text often contains error messages and other content of little or negative training value, such as navigation buttons or social media counters.[11] Raw data therefore usually goes through curation, including filtering, cleaning and compiling, and some developers rely on datasets first collected and curated by third parties.[2]
GPT-3’s developers described three steps: they filtered a version of Common Crawl by similarity to high-quality reference texts, removed near-duplicate documents, and added known high-quality datasets to the mix.[12] What goes into the data shows up in the output: the authors of the TruthfulQA benchmark found models repeating popular misconceptions, which they attributed to learning from the training distribution.[13]
Licensing, user data and synthetic data
Not all data is scraped. The Copyright Office says developers may also use licensed or non-public data: some have access to data from interactions with their customers or users, and some license data from publishers and specialized data providers.[3] It cites a July 2023 deal in which the Associated Press licensed OpenAI to train on portions of its news archive.[14] An OpenAI help page quoted in the report says ChatGPT improves by further training on users’ conversations unless they opt out.[15] The report also notes that some developers have turned to pirate sources, such as shadow libraries of published books.[16]
Synthetic data is another way to reduce reliance on large collections of human-written material.[4] The Copyright Office records concerns that using it may lead to a phenomenon called model collapse.[4]
Opting out with robots.txt
The main tool site owners have is a file called robots.txt. The Robots Exclusion Protocol, first defined in 1994 and standardized by the IETF as RFC 9309 in 2022, lets site owners say how automated crawlers may access their content.[17] Each crawler identifies itself with a product token and is expected to obey the group of allow and disallow rules that matches its name.[18]
AI-related crawlers have their own tokens:
- CCBot is Common Crawl’s crawler; a robots.txt rule for CCBot blocks it, though Common Crawl warns that some crawlers falsely claim to be CCBot.[19]
- GPTBot is OpenAI’s training crawler. OpenAI says disallowing it signals that a site’s content should not be used to train its foundation models, separately from OAI-SearchBot, which surfaces sites in ChatGPT search.[20]
- Google-Extended lets publishers control whether content Google crawls may be used to train future Gemini models; Google says it does not affect Search inclusion or ranking.[21]
There is an important limit. RFC 9309 states that robots.txt rules are not a form of access authorization and are no substitute for real security measures; crawlers are requested, not forced, to comply.[5]
The legal debate
In May 2025 the Copyright Office reported that dozens of lawsuits were pending in the United States over how copyright’s fair use doctrine applies to AI training.[6] In its report, the Office said various training uses are likely transformative, but that commercial use of vast troves of works to produce competing content, especially through illegal access, goes beyond established fair use.[22] In June 2025, two US federal judges ruled in separate cases that using authors’ books to train AI models was fair use on the records before them.[23][24]
The Office also reported that copyright owners rejected opt-out systems, and it concluded that requiring owners to opt out conflicts with the principle that their consent is required, while noting opt-outs could preserve some control if Congress ever created an AI training exception.[25]
The European Union takes a different route. The EU AI Act requires providers of general-purpose AI models to have copyright policies and to identify and respect copyright owners’ text-and-data-mining opt-outs.[26] On 24 July 2025 the European Commission published a template for the public summary of training content those providers must publish.[27]
Read next
Anthropic
Anthropic is the US AI company that makes the Claude models. Its structure, funding, 2026 IPO filings and Pentagon dispute.
Questions readers ask
Where does AI training data come from?
The US Copyright Office says a common practice is downloading publicly available data from the internet with automated tools, and that text scraped by web crawlers, often obtained from Common Crawl, is a particularly common source. Developers also license content and create synthetic data.[1][14][4]
What is Common Crawl?
An archive of web crawl data holding petabytes of raw pages, metadata and text extracts collected regularly since 2008. Access to the copy hosted on Amazon Web Services is free.[9]
Can I stop AI companies from training on my website?
You can add robots.txt rules for specific crawlers, such as CCBot for Common Crawl or GPTBot for OpenAI, and Google offers a Google-Extended token for Gemini training. Robots.txt is a request that crawlers are asked to honor, not a security control.[19][20][21][5]
Does blocking Google-Extended remove my site from Google Search?
Google says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal.[21]
Sources
· 27 claims from 12 publishers
Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.
- [1]
The US Copyright Office says training data is acquired in various ways; a common practice is downloading publicly available data from the internet with automated scraping tools, and one particularly common source is text scraped by web crawlers, often obtained from Common Crawl. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Section II.B.2 (Acquisition and Curation), PDF p. 17 (retrieved 2026-11-07)
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Acquisition and Curation, PDF p. 18 (retrieved 2026-11-07)
- [2]
The Copyright Office says raw data typically goes through curation, such as filtering, cleaning and compiling, and that some developers rely on datasets first collected and curated by third parties. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Acquisition and Curation, PDF p. 19 (retrieved 2026-11-07)
- [3]
The Copyright Office says developers may also use licensed or non-public data; some own or have access to data from interactions with their customers or users, and they may license data from third parties such as publishers and specialized data providers. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Acquisition and Curation, PDF p. 18 (retrieved 2026-11-07)
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Acquisition and Curation, PDF p. 18 (retrieved 2026-11-07)
- [4]
The Copyright Office describes synthetic data as another way to reduce dependence on human-authored data, while noting concerns that heavy use of it may lead to model collapse. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Training Data, PDF p. 15 (retrieved 2026-11-07)
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, footnote, PDF p. 15 (retrieved 2026-11-07)
- [5]
RFC 9309 says robots.txt rules are not a form of access authorization and that the protocol is not a substitute for real security measures; the rules are ones crawlers are requested to honor. confirmedas of 2022-09-01
- RFC 9309: Robots Exclusion Protocol · IETF (RFC Editor) · Primary source · 2022-09-01 · Section 2 (retrieved 2026-11-07)
- RFC 9309: Robots Exclusion Protocol · IETF (RFC Editor) · Primary source · 2022-09-01 · Section 3, Security Considerations (retrieved 2026-11-07)
- [6]
The Copyright Office said in May 2025 that dozens of lawsuits were pending in the United States over how copyright's fair use doctrine applies to AI training. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Introduction, PDF p. 5 (retrieved 2026-11-07)
- [7]
OpenAI, as quoted by the Copyright Office, describes pre-training as the step in which a massive amount of computing power and data is spent to teach a model the broad foundations of language, and post-training as further training on a smaller amount of carefully curated data. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Training, PDF p. 21 (retrieved 2026-11-07)
- [8]
A 2020 study found that language-model loss falls as a power law as model size, dataset size and training compute grow, with some trends spanning more than seven orders of magnitude. confirmedas of 2020-01-23
- Scaling Laws for Neural Language Models · arXiv (Kaplan et al.) · Primary source · 2020-01-23 · Abstract (retrieved 2026-10-10)
- [9]
Common Crawl's corpus contains petabytes of raw web page data, metadata and text extracts collected regularly since 2008, and access to the copy hosted on Amazon Web Services is free. confirmedas of 2026-11-07
- [10]
According to the Copyright Office, GPT-3 was trained with several datasets of scraped material, including Common Crawl and WebText, which made up 82% of the weight of its training mix. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, footnote 73, PDF p. 18 (retrieved 2026-11-07)
- [11]
The Copyright Office notes that text scraped from the internet often contains material of little training value, such as error messages, navigation buttons or social media counters, which curation can remove. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Training Data, PDF p. 15 (retrieved 2026-11-07)
- [12]
GPT-3's developers filtered a version of Common Crawl by similarity to high-quality reference corpora, removed near-duplicate documents, and added known high-quality corpora to the training mix. confirmedas of 2020-05-28
- Language Models are Few-Shot Learners (arXiv 2005.14165, full text) · arXiv · Primary source · 2020-05-28 · Section 2.2, Training Dataset, p. 8 (retrieved 2026-11-07)
- [13]
The TruthfulQA authors found that models generated false answers that mimic popular misconceptions, which they attribute to learning from the training distribution of human-written web text. confirmedas of 2021-09-08· interpretation
- TruthfulQA: Measuring How Models Mimic Human Falsehoods · arXiv · Primary source · 2021-09-08 · Abstract (retrieved 2026-11-07)
- TruthfulQA: Measuring How Models Mimic Human Falsehoods · arXiv · Primary source · 2021-09-08 · Abstract (retrieved 2026-11-07)
- [14]
The Copyright Office report cites a July 2023 deal in which the Associated Press licensed OpenAI to train its models on portions of the AP's text archive, as an example of licensing for AI training. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, footnote on licensing market, PDF p. 71 (retrieved 2026-11-07)
- [15]
A help page from OpenAI, quoted in the Copyright Office report, says ChatGPT improves by further training on the conversations people have with it unless they opt out. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, footnote 76, PDF p. 18 (retrieved 2026-11-07)
- [16]
The Copyright Office reports that some developers have turned to pirate sources, such as shadow libraries holding large collections of published books. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Acquisition and Curation, PDF p. 18 (retrieved 2026-11-07)
- [17]
RFC 9309, published by the IETF in September 2022, standardizes the Robots Exclusion Protocol first defined in 1994, which lets site owners use a robots.txt file to say how automated crawlers may access their content. confirmedas of 2022-09-01
- RFC 9309: Robots Exclusion Protocol · IETF (RFC Editor) · Primary source · 2022-09-01 · Abstract (retrieved 2026-11-07)
- [18]
Under RFC 9309, crawlers identify themselves with a product token and obey the group of allow and disallow rules that matches that token in a site's robots.txt file. confirmedas of 2022-09-01
- RFC 9309: Robots Exclusion Protocol · IETF (RFC Editor) · Primary source · 2022-09-01 · Section 2.2.1, The User-Agent Line (retrieved 2026-11-07)
- [19]
Common Crawl's crawler identifies itself as CCBot, and site owners can block it by adding a robots.txt rule for the CCBot user agent; Common Crawl warns that some crawlers falsely identify themselves as CCBot. confirmedas of 2026-11-07
- [20]
OpenAI says disallowing its GPTBot crawler signals that a site's content should not be used to train its generative AI foundation models, and that this setting is independent of OAI-SearchBot, which surfaces sites in ChatGPT search. confirmedas of 2026-11-07
- Overview of OpenAI Crawlers · OpenAI · Primary source · Overview of OpenAI Crawlers, GPTBot (retrieved 2026-11-07)
- Overview of OpenAI Crawlers · OpenAI · Primary source · Overview of OpenAI Crawlers (retrieved 2026-11-07)
- [21]
Google offers a robots.txt product token, Google-Extended, that lets publishers control whether content Google crawls may be used to train future Gemini models; Google says it does not affect inclusion or ranking in Google Search. confirmedas of 2026-11-07
- Google's common crawlers · Google for Developers (Google Search Central) · Primary source · Google's common crawlers, Google-Extended (retrieved 2026-11-07)
- Google's common crawlers · Google for Developers (Google Search Central) · Primary source · Google's common crawlers, Google-Extended (retrieved 2026-11-07)
- [22]
In the Part 3 pre-publication report, the Copyright Office said that various AI training uses are likely transformative, but that commercial use of vast troves of copyrighted works to produce expressive content competing with them in existing markets, especially through illegal access, goes beyond established fair use boundaries. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 pre-publication version, Conclusion (retrieved 2026-11-07)
- [23]
On 23 June 2025, Judge William Alsup granted Anthropic summary judgment that using the plaintiffs' books to train its models was fair use, and that converting purchased print books to digital form was also fair use. confirmedas of 2025-06-23
- Bartz v. Anthropic PBC, No. C 24-05417 WHA, Order on Fair Use (Dkt. 231) · US District Court for the Northern District of California (via CourtListener RECAP) · Primary source · 2025-06-23 · Order on Fair Use (Dkt. 231, case 3:24-cv-05417-WHA), Conclusion (retrieved 2026-11-07)
- [24]
On 25 June 2025, Judge Vince Chhabria granted Meta summary judgment in Kadrey v. Meta, a suit by 13 authors, finding that Meta's use of their books to train its Llama models was fair use on the record before him. confirmedas of 2025-06-25
- Federal judge sides with Meta in lawsuit over training AI models on copyrighted books · TechCrunch · Secondary source · 2025-06-25 (retrieved 2026-11-07)
- Judge reluctantly sides with Meta in Sarah Silverman, authors' AI copyright suit · Courthouse News Service · Secondary source · 2025-06-25 · Hand-saved copy (fetch-source blocked with 403) (retrieved 2026-11-07)
- Meta wins AI copyright case, but judge says others could bring lawsuits · CNBC · Secondary source · 2025-06-25 (retrieved 2026-11-07)
- [25]
The Copyright Office reported that copyright owners rejected the idea of an opt-out approach, and it concluded that requiring owners to opt out is inconsistent with the principle that consent is required, while noting that opt-outs could preserve some control if Congress ever created an AI training exception. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Opting Out, PDF p. 105 (retrieved 2026-11-07)
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, Conclusion on licensing, PDF p. 109 (retrieved 2026-11-07)
- [26]
The Copyright Office notes that the EU AI Act obliges general-purpose AI model providers to put in place policies to comply with EU law and to identify and comply with copyright owners' text-and-data-mining opt-outs. confirmedas of 2025-05-09
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) · US Copyright Office · Primary source · 2025-05-09 · Part 3 report, international approaches, PDF p. 81 (retrieved 2026-11-07)
- [27]
On 24 July 2025 the European Commission published a template for the public summary of training content that general-purpose AI model providers must make available, setting a common minimum baseline of information. confirmedas of 2025-07-24
- Explanatory Notice and Template for the Public Summary of Training Content for general-purpose AI models · European Commission · Primary source · 2025-07-24 · Library page, publication dated 24 July 2025 (retrieved 2026-11-07)
Revision history (2)
- Noted that the Copyright Office report is the May 2025 pre-publication version.
- Page created.
Created Nov 7, 2026. Last reviewed by an editor on Nov 8, 2026. Next scheduled review: Feb 7, 2027.
Cite this page
"Where AI training data comes from: crawling, licensing and opt-outs." ContentLora, updated Nov 14, 2026. https://contentlora.com/explain/how-ai-training-data-is-collected
Spotted an error? Suggest a correction or emailcorrections@contentlora.com.