I built an AI assistant for our docs at Handsontable, and then Grok found it. 4.5 million edge requests and 87,000 page renders in only 4 hours, all from xAI's Colossus data center in Memphis. I caught it during the scan and pushed a fix to redirect traffic to our MCP, then spent a week analyzing the logs and learning how crawlers work, what they look for, and what they ignore.

This would be interesting to read if not for the LLM prose style.

Another example of uselessness of llms.txt.