Cloudflare are launching a 'pay per crawl' service https://www.cloudflare.com/paypercrawl-signup/

So whilst AI companies are known to behave badly, there could be side effects:

1. Cloudflare may have a financial incentive if they are taking a cut from the payments 2. Possible risk that this may end up stopping every day users who have setup crawlers for their own needs.