SEO glossaryBingbot is the web crawler Microsoft uses to discover and index pages for Bing.
Bingbot is Microsoft's web crawler, the Bing equivalent of Googlebot. It follows links and sitemaps from page to page, fetches each URL, and passes what it finds to Bing's index. Pages it cannot reach do not appear in Bing, and because Bing's index also feeds other search services and Microsoft products, being crawlable by Bingbot matters beyond Bing itself.
How it works
Bingbot identifies itself with a user agent string containing bingbot/2.0. Like Googlebot, it renders pages with a modern browser engine, based on Microsoft Edge, so it can process most JavaScript. It still finds pages most reliably through plain HTML links and an up-to-date XML sitemap.
Bingbot respects robots.txt. Unlike Googlebot, it also honors the Crawl-delay directive, which asks it to wait between requests. In Bing Webmaster Tools, the Crawl Control setting lets you go further and choose when Bingbot crawls harder or more gently through the day.
Checking that a visitor is really Bingbot
Anyone can fake a user agent, and scrapers often pretend to be Bingbot to get past firewalls. To confirm a request is genuine, run a reverse DNS lookup on its IP address: real Bingbot addresses resolve to a hostname ending in search.msn.com, and a forward lookup of that hostname should return the same IP. Bing Webmaster Tools also has a Verify Bingbot tool that does this for you.
Helping Bingbot crawl a site
- Keep
robots.txtopen to the pages you want in search, and check that a security plugin or firewall is not blocking Bingbot by mistake. - Submit an XML sitemap in Bing Webmaster Tools, and keep it limited to live, indexable URLs.
- Link to every important page from somewhere on the site, using normal
<a href>links. - Use IndexNow, supported by Bing and several SEO plugins for WordPress, to announce new or changed pages instantly.
- Keep server responses fast and stable; repeated errors or timeouts make crawlers slow down.
Most of this is the same work that improves crawlability for every search engine.
Common problems
- Too many requests: on a slow shared server, an active crawl can be noticeable. Adjust Crawl Control rather than blocking the bot.
- Accidental blocking: a
Disallow: /left over from a staging site, or a firewall rule aimed at bad bots, can remove a site from Bing entirely. - Wasted crawling: endless filter and parameter URLs use up the attention Bingbot gives a site. See crawl budget.