I have been using llms.txt for that

i think its better use llms.txt for this

> Why your website is invisible to the agents visiting it.

I only _wish_ I could make a website invisible to agents. Instead now we're hostages to Cloudflare /shrug

Edit: so this isn't as "driveby-ish": the tldr; seems to be, if you want your site crawl-able, don't go overboard on the JS. This was good wisdom back then just as it is now in the current situation.

[flagged]