Book a call Free rebuild
Home/Glossary/SEO/Crawl Budget

Crawl Budget

October 1, 2022 Updated November 16, 2026 2 min read

SEO glossaryCrawl budget is how many pages a search engine will crawl on a site in a given time. It mostly matters for very large sites.

Crawl budget is the amount of attention a search engine gives a site: how many URLs its crawler can and wants to fetch over a period of time. Google describes it as the combination of two things. The crawl capacity limit is how fast Googlebot can crawl without straining the server, which rises when the site responds quickly and falls when it slows down or returns errors. Crawl demand is how much Google wants to crawl, based on how popular the URLs are and how often they change.

Who needs to care

For most sites, crawl budget is not a problem. Google's own guidance says it mainly matters for very large sites (around a million or more unique pages that change regularly), medium-sized sites of 10,000 or more pages whose content changes daily, and sites where much of the content sits in Google Search Console as "Discovered, currently not indexed". A typical WordPress business site or blog of a few hundred pages will be crawled fully without any special work.

There is no target number to aim for, and crawl budget is not a ranking factor. Being crawled more often does not by itself make a page rank higher.

What wastes crawl budget

  • Endless URL variations: faceted navigation, sorting and filter parameters, session IDs, and calendars that generate pages forever.
  • Duplicate content reachable at several URLs.
  • Soft 404s and long redirect chains, which use requests without returning useful pages.
  • Large numbers of low-value pages, such as thin tag archives or internal search results.
  • A slow or unstable server, which lowers the capacity limit for everything else.

How to manage it

  1. Check the data first. The Crawl stats report, under Settings in Google Search Console, shows requests per day, response times, and which file types and responses Googlebot sees.
  2. Block URLs that should never be crawled with robots.txt, such as internal search results and endless parameter combinations. Note that noindex does not save crawl budget, because Google has to fetch a page to see the tag.
  3. Consolidate duplicates and link internally to the canonical version of each page.
  4. Return the right status codes: 404 or 410 for pages that are gone, and a single 301 hop for pages that moved.
  5. Keep XML sitemaps clean, listing only live, indexable URLs with accurate lastmod dates.
  6. Improve server response time with good hosting and page caching.

In WordPress

WordPress can create more URLs than people expect: tag, author, and date archives, attachment pages (off by default on sites installed since WordPress 6.4), comment pagination, and ?s= search results. On older sites, most SEO plugins can turn attachment pages off, and they can noindex thin archives. On large WooCommerce stores, filter and sort parameters are usually the biggest source of wasted crawling, and are worth controlling first.

For a step-by-step approach, see our guide to Google crawl budget optimization, and the related term crawlability.

Source: Google Search Central, Large site owner's guide to managing your crawl budget.

WP Services

WordPress services at revolutionary pricing.

Productized WordPress services, from builds to care, delivered by Seahawk and tracked in one app. For the WordPress lovers.