Added an llms.txt to three sites last year when everyone was talking about it. Checked server logs for September:
– site 1: 4 requests, all from what looked like SEO tools
– site 2: 0
– site 3: 11, mostly unknown user agents
Meanwhile the big AI crawlers are hitting normal pages thousands of times a month and ignoring the file entirely as far as I can tell.
Not saying it’s harmful, it took 20 minutes. But I keep seeing it in audits as a recommendation like it’s a must-have.
Is anyone seeing anything in their logs that suggests it’s being used? Or should we stop putting it in audit checklists?
.
