| Important URL blocked by robots.txt |
Crawlers are instructed not to request a page that should be discoverable |
High |
Test the URL in robots.txt and Search Console URL Inspection |
| robots.txt unavailable |
Crawlers cannot reliably retrieve crawling instructions |
High |
Request /robots.txt, inspect server response and crawl logs |
| robots.txt returns 5xx |
Server failure prevents reliable processing of crawl rules |
High |
Check HTTP response and server logs |
| Invalid robots.txt rule |
Directive is malformed, unsupported, or does not match intended URLs |
Medium |
Test rules against example URLs |
| Important CSS or JS blocked |
Rendering resources needed to understand a page cannot be requested |
Medium–High |
Compare rendered page and blocked-resource reports |
| Important page not internally discoverable |
No normal crawl path leads to the page |
High |
Crawl from the homepage and compare with sitemap/GSC URLs |
| Noncrawlable JavaScript links |
Navigation acts like links for users but lacks crawlable <a href> destinations |
High |
Inspect rendered HTML and link markup |
| Important page buried too deeply |
Valuable page requires many navigation steps to reach |
Medium–High |
Measure crawl depth and inspect hierarchy |
| Crawl trap |
URLs can be generated nearly infinitely |
High |
Review crawl patterns, parameters and log files |
| Faceted navigation crawl explosion |
Filters create very large numbers of crawlable combinations |
High on large sites |
Crawl parameter patterns and compare indexable inventory |
| Internal search results crawlable |
Search queries generate discoverable URLs with little independent value |
Medium–High |
Search crawl and index data for internal-search patterns |
| Calendar crawl trap |
Forward/backward calendar navigation creates unlimited dated URLs |
Medium–High |
Inspect crawler URL patterns |
| Session IDs in URLs |
Individual sessions generate alternate versions of pages |
Medium |
Look for recurring session parameters |
| Tracking parameters crawlable |
Analytics parameters generate duplicate crawl paths |
Medium |
Segment URLs containing UTM or tracking parameters |
| Sort parameters crawlable |
Sorting changes create additional URLs without new content value |
Medium |
Compare content and canonical handling |
| Filter parameters crawlable |
Filter combinations multiply URL inventory |
Medium–High |
Crawl filters and inspect canonical/index directives |
| Important URL discovered but rarely crawled |
Google knows the URL but does not request it often enough to process changes |
Context–High |
URL Inspection, Crawl Stats and logs |
| Sudden Googlebot crawl decline |
Search engine crawling falls unexpectedly across important sections |
Context–High |
Compare Crawl Stats and server logs over time |
| Sudden crawl spike |
Googlebot begins requesting a large number of unexpected URLs |
Context |
Identify patterns in Crawl Stats/logs |
| Low-value URLs dominate crawler requests |
Crawl resources are spent on filters, parameters, search pages or duplicates |
High on large sites |
Analyze logs by URL pattern |
| XML sitemap unavailable |
Search engines cannot access the submitted sitemap |
Medium–High |
Fetch sitemap and inspect Search Console status |
| XML sitemap malformed |
XML or sitemap syntax prevents reliable processing |
Medium |
Validate sitemap |
| Sitemap contains redirects |
Sitemap asks crawlers to process URLs that immediately move elsewhere |
Medium |
Crawl all sitemap URLs and segment 3xx responses |
| Sitemap contains 4xx URLs |
Removed URLs remain submitted for indexing |
Medium |
Crawl sitemap URLs |
| Sitemap contains 5xx URLs |
Submitted URLs fail at the server |
High if widespread |
Crawl sitemap and inspect server errors |
| Sitemap contains noindex URLs |
Sitemap and indexing directives send conflicting signals |
Medium |
Compare sitemap URLs with robots directives |
| Sitemap contains noncanonical URLs |
Submitted URLs are not the preferred versions |
Medium |
Compare sitemap URLs with canonical targets |
| Sitemap omits priority canonical pages |
Important pages receive one fewer discovery/canonical signal |
Medium |
Compare indexable crawl inventory with sitemap |
Sitemap lastmod is unreliable |
Dates change without actual page changes |
Low–Medium |
Compare modification dates against real publishing history |