Book a call Free rebuild
Home/Glossary/SEO/Web Crawler

Web Crawler

October 31, 2022 Updated November 16, 2026 2 min read

SEO glossaryA web crawler, or spider, is a bot that follows links to discover and download web pages, for a search engine's index or another purpose.

A web crawler, also called a spider or bot, is a program that browses the web automatically. It starts from a list of known URLs, downloads each page, extracts the links it finds, and adds new URLs to its queue, so it keeps discovering pages without anyone pointing it to them. Search engines are the best-known users: Google's crawler is Googlebot and Microsoft's is Bingbot.

How crawling works

  1. Discovery. The crawler finds URLs through links on pages it already knows, XML sitemaps, and, for some engines, direct notifications such as IndexNow.
  2. Permission. Before fetching pages, a well-behaved crawler reads the site's robots.txt file and skips any paths it is told not to crawl.
  3. Fetching. It downloads the page, paying attention to server responses. Slow responses or errors make it crawl more slowly to avoid overloading the site.
  4. Rendering. Modern search crawlers run JavaScript in a browser engine to see content that scripts add to the page. Googlebot renders pages with a recent version of Chromium.
  5. Revisiting. Pages are crawled again over time, more often for pages that change frequently or matter more.

Crawling is only the first step. After a page is fetched, the search engine decides whether to store it in its index, and only indexed pages can rank. A page that cannot be crawled almost never ranks, which is why crawlability is the foundation of technical SEO.

Types of crawlers

  • Search engine crawlers, such as Googlebot, Bingbot, Applebot, and YandexBot, build search indexes.
  • AI crawlers, such as OpenAI's GPTBot, Anthropic's ClaudeBot, and Common Crawl's CCBot, collect pages for AI models and AI search. Most identify themselves and honor robots.txt, so site owners can choose whether to allow them.
  • SEO tool crawlers, such as AhrefsBot and SemrushBot, map links and pages across the web for SEO software.
  • Site audit crawlers, such as Screaming Frog SEO Spider, which site owners run on their own sites to find broken links, redirects, and missing tags.
  • Archive and monitoring crawlers, such as the Internet Archive's, along with price comparison and uptime services.

Controlling crawlers on a WordPress site

  • robots.txt. WordPress serves a virtual robots.txt by default, and SEO plugins let you edit it. Use it to keep crawlers out of areas that never need crawling, such as internal search results, not to hide pages from search; a blocked URL can still be indexed.
  • noindex. To keep a page out of search results, let it be crawled and add a noindex robots meta tag.
  • Crawl efficiency. On large sites, clean internal linking, fewer redirect chains, and controlled filter URLs help crawlers spend their crawl budget on important pages.
  • Bad bots. Scrapers and malicious bots often ignore robots.txt and pretend to be Googlebot. A firewall or bot management service can block them; real Googlebot traffic can be verified with a reverse DNS lookup followed by a forward DNS lookup that returns the same IP address, or by checking the IP against Google's published crawler IP ranges.

Check that security plugins and CDN bot protection are not blocking legitimate search crawlers. The Crawl stats report and URL Inspection in Google Search Console show how Googlebot is getting on.

Source: Google Search Central, Googlebot.

WP Services

WordPress services at revolutionary pricing.

Productized WordPress services, from builds to care, delivered by Seahawk and tracked in one app. For the WordPress lovers.