Book a call Free rebuild
Home/Glossary/SEO/Crawlability

Crawlability

November 3, 2022 Updated November 16, 2026 2 min read

SEO glossaryCrawlability is how easily search engine bots can find and fetch a site's pages through links, sitemaps, and a robots.txt file that lets them in.

Crawlability describes whether a search engine's crawler can reach a page and fetch its content. A web crawler such as Googlebot discovers URLs by following links and reading sitemaps, then requests each one. If a page is not linked from anywhere, is blocked, or fails to load, the crawler cannot get to it, and a page that is never crawled cannot be indexed or ranked.

Crawlability is the first step of three. Crawling fetches the page, indexing stores and understands it, and ranking decides where it appears. A page can be crawlable but still not indexed, for example when it carries a noindex tag or Google decides it is a duplicate.

What affects crawlability

  • Internal links. Crawlers follow links, so every important page needs at least one link from another page. Pages with no internal links are called orphan pages.
  • How links are coded. Google can reliably follow only <a> elements with an href attribute. Links that work only through JavaScript click handlers may never be found.
  • robots.txt. A Disallow rule tells crawlers not to fetch matching URLs. One wrong line can block a whole site.
  • XML sitemaps, which list the URLs you want crawled and help crawlers find pages that are deep in the site.
  • Server health. Server errors (5xx), timeouts, and very slow responses make crawlers back off.
  • Redirect chains and loops, which waste requests or never resolve.
  • Login walls and forms. Crawlers do not log in or submit forms, so content behind them is invisible to search.
  • Site depth. Pages many clicks from the home page are found and recrawled less often.

How to check it

  1. Open the URL Inspection tool in Google Search Console and test a live URL to see whether Google can fetch and render it.
  2. Review the Pages report for URLs that are "Blocked by robots.txt", returning errors, or "Discovered, currently not indexed".
  3. Read your robots.txt file at /robots.txt and check that nothing important is disallowed.
  4. Run a site crawler to find broken links, orphan pages, redirect chains, and very deep pages.

Crawlability in WordPress

The most common WordPress mistake is leaving "Discourage search engines from indexing this site" checked under Settings, Reading after a site launches. It adds a noindex instruction to every page. Other things to check are security plugins or firewall rules that block legitimate crawlers, and a robots.txt copied over from a staging site. WordPress and most SEO plugins create XML sitemaps automatically; make sure yours is submitted in Search Console.

On very large sites, crawlability shades into crawl budget: Google may not crawl everything it can reach. Our guide to fixing crawlability problems goes through common fixes in more detail, and crawl error explains the errors you are likely to see.

WP Services

WordPress services at revolutionary pricing.

Productized WordPress services, from builds to care, delivered by Seahawk and tracked in one app. For the WordPress lovers.