We built something quite similiar with data spanning well over 90 days. We also built a discovery index which shows how long does it take for each bot to hit the site and how many pages published in a specific time period were actually cralwed.

It shows interesting things like most aggressive crawler, most aggressive bots and so on...

We verify the bots based on reverse DNS , IP Ranges lookup and so on...

Discover index - https://mojodojo.io/ai-discovery-index Crawler Index - https://mojodojo.io/ai-crawler-index

[flagged]