It's the law of monetization.
And despite this, modern life is made possible by the illusion that "regulations" work..
If I remember correctly this "standard" was setup by someone but without involvement of any of the major AI players.
All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.
But perhaps these are developers specifically targeting these pages to feed whatever LLM they are using.
But nah, I'm sure OP doesn't know about CDNs.
So it get even stranger, I am the only one reading those /llms.txt ...
https://developers.openai.com/llms.txt
https://docs.anthropic.com/llms.txt
https://geminicli.com/llms.txt
https://github.com/llms.txt
https://docs.aws.amazon.com/llms.txt
https://openrouter.ai/docs/llms.txt[0] - https://github.com/solumos/md-browse
[1] - https://docs.stripe.com
[2] - https://vercel.com/docs
anyone who's, even slightly, clued into how agents access documentation, has been making changes to their pages. ex: https://searchtxt-web.fly.dev/search?q=aws
https://github.com/AnswerDotAI/llms-txt/issues/2
It’s been completely ignored ever since.
There is an enshittification cycle at work. The web used to be good, predominately text, and useful, 25 years ago. Then... slowly... we added javascript, then AJAX, CSS, flash, interstitials, popups, marketing, social media, algorithms, doomscrolling... gradually but surely turn it into the unusable cesspool that it is today.
Now we have AI! I think a big part of its utility is that it gets us back to text/information, and lets us bypass all the "beautiful" design / nonsense on the material it is trained on.
However, AI is just beginning its enshittification cycle - now that it has a critical mass of users, it is an irresistible target to start slowly adding ads, misinformation, conspiracy theories, and whatever else people can dream up, until it also becomes unusable and the cycle repeats.
This is just a redeux of the early web.
There are also clear definition of the restful scheme and API/data access options.
One very basic example would be the weather channel https://weather.com/llms.txt
https://news.ycombinator.com/item?id=48411569
BTW why should Chrome even consider rendering a .txt file as markdown?
Result: no such item.
From where do you got the idea that adding /llm.txt to urls will produce markdown?
I imagine Claude could zero-shot a Chrome plugin for that.