Is crawl budget a real concern for a 40k URL site or am I overthinking it?

› Forums › Digital Marketing › Seo › Is crawl budget a real concern for a 40k URL site or am I overthinking it?

  • Post
    • September 25, 2026 at 1:29 am
    crawlbudget_carl
    Participant
    Running SEO for a mid-size home goods store, roughly 40k indexable URLs plus a lot of parameter junk. Log files show Googlebot hitting about 6-7k URLs a day, and maybe a third of those are sort/filter parameters we never wanted crawled.

    New products take 2-3 weeks to show up in the index, which is what got me looking at this. Everyone says crawl budget only matters for sites with millions of pages, but the numbers above don’t feel great either.

    Would you block the parameters in robots.txt, or just clean up internal links and let Google figure it out?

Viewing 1 replies (of 1 total)
  • Replies
    • September 25, 2026 at 2:48 am
    VladTheIndexer
    Participant
    Normal
    tbh a third of hits wasted on parameters is exactly the case where it matters, size aside. I’d fix the internal links first (sort links as buttons or with the param stripped), then block the worst patterns in robots once you’ve confirmed nothing valuable lives on them. Also check your XML sitemap lastmod, if it’s stamping every URL with today’s date Google learns to ignore it and new products wait longer.
Viewing 1 replies (of 1 total)
  • You must be logged in to reply to this topic.