Finally got access to raw access logs for a client (mid size ecommerce, around 8k URLs). Parsed about 30 days of Googlebot hits after verifying the IPs.
Things I found that I didn’t expect:
– about 35% of Googlebot requests went to faceted filter URLs that are canonicalised elsewhere
– a bunch of old redirected URLs from a migration 2 years ago still getting crawled daily
– some category pages deep in the nav hadn’t been hit once in 30 days
The crawl stats report in GSC kind of hinted at some of this but nowhere near this detail.
Curious what others have found from logs. Was it worth the hassle for you or is it overkill for most sites?
.
