When publishers complain it's never to protect the authors or journalists, but a specific (extractive) business model. Not that I side with Common Crawl on this one.

maybe robot.txt should be upgraded with license specifics.

not for commercial use, etc.

so I believe all content is covered under fair use which to me means common crawl has a right to scrape everything and it's the user of common crawl to sort out the details.