Book a call Free rebuild
Home/Glossary/SEO/Crawl Error

Crawl Error

January 6, 2023 Updated November 16, 2026 3 min read

SEO glossaryA crawl error happens when a search engine bot tries to reach a page and fails, for example because of a 404, a server error, or a redirect loop.

Before a page can appear in search results, a crawler such as Googlebot has to fetch it. A crawl error is any failed attempt: the server did not answer, refused access, returned an error, or sent the bot round in circles. Some crawl errors are harmless, such as a 404 for a page you deleted on purpose. Others, like a server that keeps timing out, can stop a whole site from being crawled.

Types of crawl errors

Site-wide errors

  • DNS errors: the crawler cannot resolve the domain, often after a domain expires or DNS records are changed incorrectly.
  • Server errors (5xx): the server crashed, timed out, or was overloaded. Repeated 5xx responses make Google slow its crawling.
  • robots.txt problems: if the robots.txt file itself returns a server error, Google may hold off crawling the site until it can read the file.

Page-level errors

  • 404 Not found and 410 Gone: the page does not exist. Fine for pages that are meant to be gone; a problem when important pages or internal links point there.
  • Soft 404: the page says "not found" or is nearly empty but returns a 200 OK status, so search engines have to guess.
  • 401 and 403: the crawler is asked to log in or is refused, often by a security plugin, firewall, or password-protected staging setting left on.
  • Redirect errors: redirect loops, chains that are too long, or redirects to broken URLs.

How to find crawl errors

Google Search Console no longer has a report called "Crawl errors"; the same information now lives in a few places:

  • The Page indexing report lists URLs that are not indexed and why, including 404s, soft 404s, server errors, and redirect errors.
  • The Crawl stats report (under Settings) shows host problems with DNS, server connectivity, and robots.txt, plus response codes over time.
  • The URL Inspection tool tests a single URL live and shows what Googlebot received.

A site crawler such as Screaming Frog, or a link checker, will find broken links inside your own site before search engines do. Bing Webmaster Tools has similar reports for Bingbot.

How to fix them

  1. Fix server errors first. They affect everything. Check hosting resource limits, error logs, and recent plugin or theme updates.
  2. Redirect moved pages. When a page has a new address, add a 301 redirect from the old URL to the closest matching page.
  3. Let dead pages be dead. A 404 or 410 for content that is truly gone is correct and does not hurt the rest of the site. Do not redirect every 404 to the home page; Google may treat those as soft 404s.
  4. Update internal links so they point straight to live URLs, and remove dead URLs from the XML sitemap.
  5. Unblock crawlers that a firewall or security plugin is refusing, and check robots.txt is not disallowing pages you want indexed.
  6. Validate the fix in Search Console so Google rechecks the affected URLs.

Common causes on WordPress sites

  • Sudden site-wide 404s on posts often mean the rewrite rules need refreshing: saving Settings > Permalinks without changes usually fixes it.
  • Changing a post's slug or the permalink structure without planning redirects.
  • Security plugins or a web application firewall rate-limiting or blocking Googlebot.
  • Cheap shared hosting that returns 5xx errors under load.

Crawl errors are one part of crawlability; on large sites they also waste crawl budget.

Source: How HTTP status codes affect Google's crawlers.

WP Services

WordPress services at revolutionary pricing.

Productized WordPress services, from builds to care, delivered by Seahawk and tracked in one app. For the WordPress lovers.