Time to first Googlebot crawl

What is measured

The metric starts when a URL is submitted to IndexChex and stops at Googlebot's first recorded request for that URL. The weekly benchmark reports the average across submitted URLs: under 2 minutes, in both standard and instant mode. The formal definition sits with the others on the methodology page.

This is the one step IndexChex guarantees. It commits to Googlebot crawling submitted URLs; it does not commit to indexation, and credits are not refunded when a crawled URL stays out of the index.

Why it matters

Google's documentation describes search in three stages: crawling, indexing and serving. A page that is never fetched cannot be indexed, and most backlinks sit on pages that Google might otherwise take days or weeks to revisit, especially deep pages on large or low-traffic hosts. Shortening the wait for the first fetch removes the discovery delay, which is the part an indexer can control.

What it does not remove is the evaluation that follows. A URL fetched in 90 seconds can still be judged thin, duplicate or unhelpful. That is why the observatory reports first crawl separately from indexation and covers the space between them on crawl-to-index lag.

Same crawl, different outcomes

The two modes share the under-2-minute average to first crawl but differ sharply afterwards:

The table shows that crawl speed is not what separates the modes. Readers comparing tools on crawl times alone are comparing the first step of a longer process. See instant-mode rate and standard-mode rate for each path.

What can delay the first crawl

The average assumes URLs Googlebot is allowed to fetch. Several conditions push the time up or prevent a crawl altogether:

• robots.txt disallow on the path. Googlebot will not fetch it.
• Server errors or timeouts at the moment of the request.
• Firewall or bot-protection rules that challenge or block Google's crawlers.
• Redirect chains that end on a different URL, so the submitted URL itself is never the final fetch.

These are properties of the source site. A crawl trigger cannot override a block.

Checking a crawl yourself

Owners of the source site can confirm a crawl in access logs. Google publishes how to verify that a request genuinely comes from Googlebot: a reverse DNS lookup on the requesting IP should resolve to a Google crawler hostname, and a forward lookup on that hostname should return the same IP. User-agent strings alone can be spoofed. Backlinks on third-party sites usually offer no log access, which is why the benchmark's later steps rely on index checks rather than crawl logs. A full test plan, including a control group, is on how to benchmark an indexer yourself.

Cost context

A crawl is bought with the same credit as the submission, so the cost of reaching first crawl is the submission price: $0.0014 to $0.0044 per URL in standard mode, roughly $0.084 to $0.26 in instant mode. Because the indexation rate is below 100%, the cost per indexed URL is higher, up to about $0.0052 in standard mode; the arithmetic is shown on the mode pages.

Compared with vendor statements

GigaIndexer states a median bot hit of 47 seconds on its home page (checked 2026-10-08). That is a median rather than an average, it is a vendor statement rather than a test, and it describes a crawl, not indexation. IndexBolt's pricing page states standard crawling under 6 hours and instant under 1 hour. Neither is directly comparable to the IndexChex figure without the same measurement method; more on that in vendor-published indexing claims.