When I just want to see what bots are doing on a site this week, I run GoAccess on the access log instead of loading everything into a big log analyser. It is free, open source and runs in the terminal or outputs an HTML report.
How I use it:
– Grep the log for Googlebot first, then pipe that into GoAccess. You get top URLs crawled, status codes and hits per day in seconds.
– Watch the 404 and 301 sections. If Googlebot is burning hits on redirected URLs, your internal links need fixing.
– Use the real-time HTML output during a migration so you can see status codes change live.
– Set the log format correctly once in the config file, otherwise the numbers are garbage.
Downside: it does not verify bots, so fake Googlebot user agents get counted too. Do a reverse DNS check on the top IPs before drawing conclusions.
https://goaccess.io/
.
