Book a call Free rebuild
Home/Glossary/SEO/Search Index

Search Index

August 10, 2022 Updated November 16, 2026 2 min read

SEO glossaryA search index is the database of pages a search engine has crawled and stored. Only pages in the index can appear in results.

A search index is what a search engine actually searches. When you type a query into Google, it does not scan the live web; it looks up its own enormous, pre-built database of pages it has already found, read, and stored. Google describes its index as a large database hosted on thousands of computers. If a page is not in the index, it cannot appear in results, however good it is.

Like the index at the back of a book, a search index is organized for fast lookup: it records which words and concepts appear on which pages, along with information about each page, so that results can be returned in a fraction of a second.

How pages get into the index

Google describes three stages, and not every page makes it through all of them:

  1. Crawling. A web crawler such as Googlebot discovers URLs, mostly by following links and reading XML sitemaps, and downloads the pages.
  2. Indexing. Google analyzes the page's text, title, images, video, and other elements. It groups duplicate or very similar pages and picks one as the canonical version to show in results. It also records signals such as the page's language and the country it targets.
  3. Serving. When someone searches, Google pulls matching pages from the index and ranks them by relevance and quality.

Indexing is not guaranteed. Google may crawl a page and still choose not to index it, typically because the content is low quality or duplicated, because a robots meta tag tells it not to, or because the site's design makes the page hard to process.

How to check whether a page is indexed

  • URL Inspection in Google Search Console shows whether a specific URL is indexed, which canonical Google chose, and any problems. It also lets you request indexing.
  • The Page indexing report lists indexed and non-indexed pages across the site, with the reason each page was left out.
  • A site:example.com search gives a rough idea, but its counts are estimates and it is not a reliable diagnostic.

Keeping pages out of the index

Not every page should be indexed. Thank-you pages, internal search results, and staging sites are common examples. A noindex robots meta tag or HTTP header is the reliable way to keep a page out. Blocking it in robots.txt is not: that stops crawling, but a blocked URL can still be indexed without its content if other pages link to it, and Google cannot see a noindex tag on a page it is not allowed to crawl.

Indexing and WordPress

  • The setting Settings > Reading > Search engine visibility asks search engines not to index the whole site. It is meant for sites under construction and is a common cause of a live site vanishing from Google when it is left on after launch.
  • WordPress generates an XML sitemap at /wp-sitemap.xml, and SEO plugins typically replace it with their own. Submit the sitemap in Search Console.
  • SEO plugins let you noindex thin archive pages, such as tag or attachment pages, that add little on their own.

Related terms: crawlability, crawl budget, canonicalization, and de-indexing.

Source: Google Search Central: In-depth guide to how Google Search works.

WP Services

WordPress services at revolutionary pricing.

Productized WordPress services, from builds to care, delivered by Seahawk and tracked in one app. For the WordPress lovers.