Skip to content

Course highlights · Lesson 1 of 5

How Google and AI search engines find, read, rank and cite a page

By · Published · 4 min read

Google and AI search engines take every page through four steps: they find it, read it, rank it and cite it. A search engine is a system that finds, stores and ranks web pages; an AI search engine also writes an answer and cites pages as its sources. Google documents the first three steps as crawling, indexing and serving.

Key takeaways on how search engines work

  • Search engines find, read, rank and cite a page, and each step has its own check.
  • Google's minimums are simple: Googlebot can reach the page, the page returns HTTP 200 and it has indexable content.
  • AI search engines use named crawlers, many read only server HTML, and they favor content that is easy to quote and verify.
Four steps from page to answer: found through links and sitemaps, read as HTML, ranked by signals, cited when quotable
How search engines and AI search engines handle a page: they find, read, rank and cite it, in that order.

How does Google find and store a page?

Google finds a page by crawling: its crawler, Googlebot, follows links from pages it already knows and reads the sitemaps site owners submit. During the crawl, Google renders the page with a recent version of Chrome and runs its JavaScript. In indexing, Google analyzes the page and stores what it learned in the Google index. Google's guide to how Search works states that indexing is not guaranteed, and its ranking systems then sort through hundreds of billions of pages to show the most useful results. Google also states that it does not accept payment to crawl a site more often or to rank it higher.

Google's technical requirements list three minimums for a page:

  • robots.txt and other rules let Googlebot reach the page.
  • The page works: it returns an HTTP 200 (success) status code.
  • The page has indexable content.

How do AI search engines fetch pages?

AI search engines fetch pages in two ways: search crawlers build an index ahead of time, and user-triggered fetchers open a page live when someone asks a question. Each company documents its own user agents, and robots.txt can name each one.

User agents documented by AI search engines
CompanySearch crawlerUser-triggered fetcher
OpenAI (ChatGPT)OAI-SearchBotChatGPT-User
Anthropic (Claude)Claude-SearchBotClaude-User
PerplexityPerplexityBotPerplexity-User

Why does server HTML matter for AI crawlers?

Server HTML matters because many AI crawlers read only the HTML a server sends and do not run JavaScript. A 2024 analysis by Vercel found that the crawlers behind ChatGPT and Claude fetched JavaScript files but did not execute them. Google renders JavaScript in a separate step, and a page can wait in its rendering queue. Main text in the server HTML reaches every reader from the first request.

What makes a page easy to cite?

A page is easy to cite when it is indexed, eligible for a snippet, and easy to quote and verify. Google explains that AI Overviews and AI Mode may use query fan-out, running several related searches across subtopics to build one response, so one clear section can answer a narrow sub-question and become the linked source. A nosnippet rule or a very short max-snippet limit reduces what Google can show from the page. In the GEO (Generative Engine Optimization) study by Aggarwal and colleagues, citing sources, adding quotations and adding statistics worked best, while keyword stuffing offered little to no improvement.

Which four questions should you ask about any page?

Four questions, one per step, show where a page needs work:

  1. Can it be found? The page is linked from another page, listed in the sitemap and allowed in robots.txt.
  2. Can it be read without scripts? The main text is in the HTML the server sends.
  3. Does it deserve to rank? It answers the search better than the alternatives.
  4. Is it easy to quote? Each section answers one question, with a source for each claim.

Questions about how search engines work

Does meeting Google's technical requirements guarantee indexing?
No. Google states that indexing is not guaranteed. Meeting the three minimums lets a page be considered; quality and usefulness decide the rest.
Do AI search engines run JavaScript?
Many do not. A 2024 Vercel analysis found that the crawlers behind ChatGPT and Claude fetched JavaScript files without executing them, so main content belongs in the server HTML.
How can I see what Google received from my page?
Use the URL Inspection tool in Google Search Console. It shows whether a page is indexed, and its live test shows the HTML Google received and a screenshot.

Sources for this lesson on how search engines work

  1. In-depth guide to how Google Search works (Google Search Central)
  2. Google Search technical requirements (Google Search Central)
  3. Understand JavaScript SEO basics (Google Search Central)
  4. A guide to Google Search ranking systems (Google Search Central)
  5. AI features and your website (Google Search Central)
  6. Overview of OpenAI crawlers (OpenAI)
  7. Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic)
  8. Perplexity crawlers (Perplexity)
  9. The rise of the AI crawler (Vercel, 17 Dec 2024)
  10. GEO: Generative Engine Optimization (Aggarwal et al., 2024)
  11. URL Inspection tool (Search Console Help)

Free website audit

Check this on your site with a free ContentMas scan

One score, six pillar scores and a ranked fix list in a few minutes. No credit card.

Start free audit