Explainer
How search engines work
A search engine works in three stages: crawlers download pages, an index stores what they found, and ranking systems pick the most relevant results for each query.[1][2] Google says it ranks pages programmatically using hundreds of factors, from links between pages to AI language models, and does not accept payment to rank pages higher.[3][4][5] In 2025 a US court barred Google from exclusive search distribution deals and ordered it to share some search data with rivals; as of October 2026, both sides had appealed.[6][7]
This explainer uses Google’s own documentation as its main source because Google publishes the most detail about how its search engine works. For how ads appear next to results, see how online advertising works. Choose a reading level in each section.
Step 1: crawling
The web has no master list of pages. Search engines have to find them.[8] One way Google does this is by following links from pages it already knows. Site owners can also send it a list of their pages, called a sitemap.[8]
A program called a crawler then visits each page and downloads it. Google’s crawler is called Googlebot. It decides which sites to visit, how often and how many pages to fetch. It also tries not to visit a site so fast that it slows the site down.[9]
Site owners can put a file called robots.txt on their site to tell crawlers which pages they may visit. This does not hide a page from search results. For that, Google says a page needs a “noindex” rule or a password.[10]
Google calls the first step “URL discovery”: known pages are revisited, new URLs are extracted from links on known pages, and sitemaps add more.[8] Googlebot’s crawl scheduling is algorithmic (which sites, how often, how many pages), and the crawler renders each page with a recent version of Chrome so that content injected by JavaScript is visible.[9]
robots.txt controls crawler access, not indexing. Google says it exists mainly to avoid overloading sites with requests, and that keeping a page out of Google needs noindex or password protection.[10]
Step 2: indexing
After a page is downloaded, the search engine works out what it is about. Google reads the text, titles, images and videos on the page.[11] It stores what it learns in the index, a huge database spread over thousands of computers.[2]
The web is full of copies. When several pages say much the same thing, Google groups them and picks one to show, called the canonical page.[11] Not every page makes it into the index.[2]
Indexing analyses textual content and key tags and attributes, such as <title> elements and alt text, along with images and video.[11] Duplicate handling happens here: Google clusters pages with similar content and selects the most representative as canonical, the version that may be shown in results.[11] The resulting index is stored across thousands of machines, and Google states plainly that indexing is not guaranteed.[2]
Step 3: ranking the results
When you type a search, the engine looks in its index for matching pages and puts them in order. Google says it uses hundreds of factors. Some depend on you, such as where you are, your language and whether you are on a phone or a computer.[3]
One of the oldest factors is links. Google’s PageRank system looks at how pages link to each other, and it is still part of how results are ranked.[4] Google also uses AI to understand what words mean, so it can find a useful page even if it does not contain the exact words you typed.[5] Google says it does not take money to rank pages higher.[3]
Google describes its ranking as automated systems weighing many signals across hundreds of billions of documents in a fraction of a second.[12] Its published list includes link analysis (PageRank, used since launch and since much evolved), and language-understanding models: BERT for how word combinations express meaning, RankBrain for relating words to concepts, and neural matching for matching concept representations in queries and pages.[4][5] Specialised systems handle “query deserves freshness” cases and deduplicate near-identical results.[13] Context such as location, language and device also feeds relevance.[3] BERT stands for Bidirectional Encoder Representations from Transformers.[5] For transformers, see how large language models work.
Fighting spam
Because a top spot in search is valuable, some sites try to trick search engines. Google bans tricks such as showing search engines a different page from the one people see, creating links just to push up rankings, and churning out large numbers of low-value pages, including with AI tools.[14] It uses automated systems, including one called SpamBrain, to catch them.[14]
Google’s spam policies define cloaking (serving different content to users and crawlers to manipulate rankings), link spam (links created primarily to manipulate rankings) and scaled content abuse, which explicitly covers mass-producing pages with generative AI without adding value.[14] Enforcement is largely automated through detection systems such as SpamBrain.[14]
AI answers in search
Google also offers AI-written answers in search, called AI Overviews and AI Mode. Both include links to the websites they draw on.[15] To write an answer, they may run several related searches behind the scenes.[15] For what an AI answer costs in electricity, see how much energy an AI query uses.
Google says AI Overviews and AI Mode may use “query fan-out”, issuing multiple related searches across subtopics and data sources, then cite supporting pages.[15] It tells publishers there are no additional technical requirements to appear in them beyond ordinary SEO practice.[15]
Competition and the law
The US Justice Department sued Google over search in 2020, and a court ruled against Google in August 2024.[16] In September 2025 the judge banned Google from exclusive deals to distribute its search engine and some other apps, and ordered it to share some of its search data with rivals.[6] As of October 2026, Google, the Justice Department and the states had all appealed.[7]
The October 2020 case, joined by 11 state attorneys general, produced a liability ruling in August 2024 and a final judgment in December 2025.[16] The September 2025 remedies bar exclusive contracts covering Search, Chrome, Google Assistant and the Gemini app, and require Google to make certain index and user-interaction data available to rivals and to offer them search and search-text-ads syndication.[6] Google appealed in January 2026 and the Justice Department and states also appealed in February; Alphabet said the appeals were pending as of October 2026.[7]
Read next
How recommendation algorithms work
How feeds on YouTube, TikTok and Netflix pick what to show you: candidate generation, collaborative filtering, ranking signals, filter bubbles and EU rules.
Questions readers ask
How does Google find new web pages?
There is no central registry of web pages. Google discovers new pages in several ways, including following links from pages it already knows and sitemaps that site owners submit, then sends its crawler, Googlebot, to fetch them.[8][9]
Does robots.txt keep a page out of Google?
No. Google says robots.txt tells crawlers which URLs they may access and is mainly used to avoid overloading a site. To keep a page out of Google, it says to use a noindex rule or password protection.[10]
Can you pay Google to rank higher in search results?
Google says it does not accept payment to rank pages higher and that ranking is done programmatically. Ads are sold separately through an auction.[3][17]
Is every page Google crawls included in search?
No. Google says indexing is not guaranteed and not every page it processes is indexed. Where several pages are near-duplicates, it picks one canonical version to show.[2][11]
How do AI Overviews choose their sources?
Google says AI Overviews and AI Mode may run several related searches across subtopics, a "query fan-out", and show links to supporting websites. It says there are no extra requirements for appearing in them beyond normal SEO practice.[15]
What did the court order in the Google search antitrust case?
In September 2025 a federal judge barred Google from exclusive contracts for distributing Search, Chrome, Google Assistant and the Gemini app, and ordered it to share some search index and user-interaction data with rivals. Google, the Justice Department and the states all appealed in early 2026.[6][7]
Sources
· 17 claims from 4 publishers
Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.
- [1]
Google describes Search as working in three stages, crawling, indexing and serving results, and says not every page makes it through each stage. confirmedas of 2025-12-18
- In-depth guide to how Google Search works · Google Search Central · Primary source · Introducing the three stages of Google Search (retrieved 2026-11-07)
- [2]
Google's index is a large database hosted on thousands of computers, and not every page Google processes is indexed. confirmedas of 2025-12-18
- In-depth guide to how Google Search works · Google Search Central · Primary source · Indexing (retrieved 2026-11-07)
- [3]
When someone searches, Google looks up matching pages in its index and ranks them programmatically using hundreds of factors, which can include the user's location, language and device; it says it does not accept payment to rank pages higher. confirmedas of 2025-12-18
- In-depth guide to how Google Search works · Google Search Central · Primary source · Serving search results (retrieved 2026-11-07)
- In-depth guide to how Google Search works · Google Search Central · Primary source · Serving search results (retrieved 2026-11-07)
- In-depth guide to how Google Search works · Google Search Central · Primary source · Serving search results (retrieved 2026-11-07)
- [4]
PageRank, a link-analysis system that uses how pages link to each other, was one of Google's core ranking systems at launch and, in evolved form, is still part of its core ranking systems. confirmedas of 2026-11-07
- A guide to Google Search ranking systems · Google Search Central · Primary source · Link analysis systems and PageRank (retrieved 2026-11-07)
- A guide to Google Search ranking systems · Google Search Central · Primary source · Link analysis systems and PageRank (retrieved 2026-11-07)
- [5]
Google uses AI systems such as BERT, RankBrain and neural matching to understand how combinations of words express meaning and how queries relate to concepts, so it can return relevant pages that do not contain all the exact search words. confirmedas of 2026-11-07
- A guide to Google Search ranking systems · Google Search Central · Primary source · BERT (retrieved 2026-11-07)
- A guide to Google Search ranking systems · Google Search Central · Primary source · RankBrain (retrieved 2026-11-07)
- A guide to Google Search ranking systems · Google Search Central · Primary source · Neural matching (retrieved 2026-11-07)
- [6]
On September 2, 2025 the court barred Google from exclusive contracts for distributing Search, Chrome, Google Assistant and the Gemini app, and ordered it to share certain search index and user-interaction data and offer search and search text ads syndication to rivals. confirmedas of 2025-09-02
- Department of Justice Wins Significant Remedies Against Google · US Department of Justice · Primary source · 2025-09-02 · First paragraph (retrieved 2026-11-07)
- Department of Justice Wins Significant Remedies Against Google · US Department of Justice · Primary source · 2025-09-02 · First paragraph (retrieved 2026-11-07)
- Alphabet Inc. Form 10-Q for the quarter ended September 30, 2026 · Alphabet (SEC filing) · Primary source · Note 10, Antitrust Matters, Search (retrieved 2026-11-07)
- [7]
According to Alphabet's filings, Google appealed the search final judgment in January 2026 and the Justice Department and states also appealed in February 2026; Alphabet said the appeals were pending as of October 2026. confirmedas of 2026-10-29
- Alphabet Inc. Form 10-Q for the quarter ended September 30, 2026 · Alphabet (SEC filing) · Primary source · Note 10, Antitrust Matters, Search (retrieved 2026-11-07)
- Alphabet Inc. Form 10-Q for the quarter ended September 30, 2026 · Alphabet (SEC filing) · Primary source · Note 10, Antitrust Matters, Search (retrieved 2026-11-07)
- [8]
There is no central registry of web pages, so Google constantly looks for new and updated pages, a process it calls URL discovery; ways it finds them include following links from pages it already knows and lists of pages (sitemaps) that site owners submit. confirmedas of 2025-12-18
- In-depth guide to how Google Search works · Google Search Central · Primary source · Crawling (retrieved 2026-11-07)
- In-depth guide to how Google Search works · Google Search Central · Primary source · Crawling (retrieved 2026-11-07)
- In-depth guide to how Google Search works · Google Search Central · Primary source · Crawling (retrieved 2026-11-07)
- [9]
Google's crawler, Googlebot, uses an algorithm to decide which sites to crawl, how often and how many pages to fetch, tries not to overload sites, and renders pages with a recent version of Chrome so it can see content added by JavaScript. confirmedas of 2025-12-18
- In-depth guide to how Google Search works · Google Search Central · Primary source · Crawling (retrieved 2026-11-07)
- In-depth guide to how Google Search works · Google Search Central · Primary source · Crawling (retrieved 2026-11-07)
- In-depth guide to how Google Search works · Google Search Central · Primary source · Crawling (retrieved 2026-11-07)
- [10]
A robots.txt file tells search engine crawlers which URLs they may access; Google says it is mainly used to avoid overloading a site and is not a way to keep a page out of Google, for which a noindex rule or password protection is needed. confirmedas of 2026-11-07
- Introduction to robots.txt · Google Search Central · Primary source · Introduction to robots.txt (retrieved 2026-11-07)
- Introduction to robots.txt · Google Search Central · Primary source · Introduction to robots.txt (retrieved 2026-11-07)
- [11]
During indexing Google analyses a page's text, key tags and attributes, images and video, groups pages with similar content and picks the most representative one, the canonical, which is the version that may be shown in results. confirmedas of 2025-12-18
- In-depth guide to how Google Search works · Google Search Central · Primary source · Indexing (retrieved 2026-11-07)
- In-depth guide to how Google Search works · Google Search Central · Primary source · Indexing (retrieved 2026-11-07)
- In-depth guide to how Google Search works · Google Search Central · Primary source · Indexing (retrieved 2026-11-07)
- [12]
Google says its automated ranking systems look at many factors and signals about hundreds of billions of web pages and other content in its index to return results in a fraction of a second. confirmedas of 2026-11-07
- A guide to Google Search ranking systems · Google Search Central · Primary source · Introduction (retrieved 2026-11-07)
- [13]
Google runs "query deserves freshness" systems that favour recent content where it would be expected, and deduplication systems that show only the most relevant of many near-identical pages. confirmedas of 2026-11-07
- A guide to Google Search ranking systems · Google Search Central · Primary source · Freshness systems (retrieved 2026-11-07)
- A guide to Google Search ranking systems · Google Search Central · Primary source · Deduplication systems (retrieved 2026-11-07)
- [14]
Google uses spam detection systems, including one called SpamBrain, against practices its policies ban, such as cloaking (showing search engines different content from users), link spam and scaled content abuse, which includes using generative AI to churn out many pages without adding value. confirmedas of 2026-11-07
- A guide to Google Search ranking systems · Google Search Central · Primary source · Spam detection systems (retrieved 2026-11-07)
- Spam policies for Google web search · Google Search Central · Primary source · Cloaking (retrieved 2026-11-07)
- Spam policies for Google web search · Google Search Central · Primary source · Link spam (retrieved 2026-11-07)
- Spam policies for Google web search · Google Search Central · Primary source · Scaled content abuse (retrieved 2026-11-07)
- [15]
Google's AI Overviews and AI Mode generate AI responses with links to supporting websites, may use a "query fan-out" technique that issues multiple related searches across subtopics, and have no extra requirements for sites beyond normal SEO best practice. confirmedas of 2026-11-07
- AI features and your website · Google Search Central · Primary source · How AI features work in Search (retrieved 2026-11-07)
- AI features and your website · Google Search Central · Primary source · How AI features work in Search (retrieved 2026-11-07)
- AI features and your website · Google Search Central · Primary source · Introduction (retrieved 2026-11-07)
- AI features and your website · Google Search Central · Primary source · How AI features work in Search (retrieved 2026-11-07)
- [16]
The US Justice Department sued Google over search in October 2020, joined by 11 state attorneys general; according to Alphabet's filings, the District of Columbia federal court ruled against Google in August 2024 and entered a final judgment in December 2025. confirmedas of 2026-10-29
- Department of Justice Wins Significant Remedies Against Google · US Department of Justice · Primary source · 2025-09-02 · Closing paragraphs (retrieved 2026-11-07)
- Alphabet Inc. Form 10-Q for the quarter ended September 30, 2026 · Alphabet (SEC filing) · Primary source · Note 10, Antitrust Matters, Search (retrieved 2026-11-07)
- [17]
Google Ads chooses which ads to show through an auction that runs every time someone searches on Google or visits a site that shows its ads, with separate auctions for each ad location on a results page. confirmedas of 2026-11-07
- About the ad auction · Google Ads Help · Primary source · How the Google Ads auction works (retrieved 2026-11-07)
- About the ad auction · Google Ads Help · Primary source · How the Google Ads auction works (retrieved 2026-11-07)
Revision history (2)
- Dated the appeal status in the Google search case (as of October 2026).
- Page created.
Created Nov 7, 2026. Last reviewed by an editor on Nov 8, 2026. Next scheduled review: May 7, 2027.
Cite this page
"How search engines work." ContentLora, updated Nov 14, 2026. https://contentlora.com/explain/how-search-engines-work
Spotted an error? Suggest a correction or emailcorrections@contentlora.com.