SEO glossaryA crawl error happens when a search engine bot tries to reach a page and fails, for example because of a 404, a server error, or a redirect loop.
Before a page can appear in search results, a crawler such as Googlebot has to fetch it. A crawl error is any failed attempt: the server did not answer, refused access, returned an error, or sent the bot round in circles. Some crawl errors are harmless, such as a 404 for a page you deleted on purpose. Others, like a server that keeps timing out, can stop a whole site from being crawled.
Types of crawl errors
Site-wide errors
- DNS errors: the crawler cannot resolve the domain, often after a domain expires or DNS records are changed incorrectly.
- Server errors (5xx): the server crashed, timed out, or was overloaded. Repeated 5xx responses make Google slow its crawling.
- robots.txt problems: if the robots.txt file itself returns a server error, Google may hold off crawling the site until it can read the file.
Page-level errors
- 404 Not found and 410 Gone: the page does not exist. Fine for pages that are meant to be gone; a problem when important pages or internal links point there.
- Soft 404: the page says "not found" or is nearly empty but returns a 200 OK status, so search engines have to guess.
- 401 and 403: the crawler is asked to log in or is refused, often by a security plugin, firewall, or password-protected staging setting left on.
- Redirect errors: redirect loops, chains that are too long, or redirects to broken URLs.
How to find crawl errors
Google Search Console no longer has a report called "Crawl errors"; the same information now lives in a few places:
- The Page indexing report lists URLs that are not indexed and why, including 404s, soft 404s, server errors, and redirect errors.
- The Crawl stats report (under Settings) shows host problems with DNS, server connectivity, and robots.txt, plus response codes over time.
- The URL Inspection tool tests a single URL live and shows what Googlebot received.
A site crawler such as Screaming Frog, or a link checker, will find broken links inside your own site before search engines do. Bing Webmaster Tools has similar reports for Bingbot.
How to fix them
- Fix server errors first. They affect everything. Check hosting resource limits, error logs, and recent plugin or theme updates.
- Redirect moved pages. When a page has a new address, add a 301 redirect from the old URL to the closest matching page.
- Let dead pages be dead. A 404 or 410 for content that is truly gone is correct and does not hurt the rest of the site. Do not redirect every 404 to the home page; Google may treat those as soft 404s.
- Update internal links so they point straight to live URLs, and remove dead URLs from the XML sitemap.
- Unblock crawlers that a firewall or security plugin is refusing, and check robots.txt is not disallowing pages you want indexed.
- Validate the fix in Search Console so Google rechecks the affected URLs.
Common causes on WordPress sites
- Sudden site-wide 404s on posts often mean the rewrite rules need refreshing: saving Settings > Permalinks without changes usually fixes it.
- Changing a post's slug or the permalink structure without planning redirects.
- Security plugins or a web application firewall rate-limiting or blocking Googlebot.
- Cheap shared hosting that returns 5xx errors under load.
Crawl errors are one part of crawlability; on large sites they also waste crawl budget.