Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.
From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirely sure but I believe google has two categories of scrapers, the regular one and a new one for AI
> Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation
Ok. So you also don't know. Well, I don't know either, but I don't make a speculation by claiming x, y, and z companies to be exempt. In my book they are all responsible.
So, it's still possible they are also running a bunch of poorly-coded scraper bots through residential proxies, but it seems a little non-obvious why they would be doing this.
(It is worth pointing out that most of this traffic seems to be dumb: it's stuff like getting lost in generated link forests of some web apps or repeatly re-querying the same endpoint on a super-high frequency. This isn't exactly going to give a good return on investment for AI training data, especially since AIUI the main race for LLM performance now is in good quality training data)
why "non obvious"? The easiest explanation is gathering training data sets, and is mostly caused by AI companies guarding their pile of essentially stolen IP (given how little they care about copyright) from eachother, without sharing any competition need to get their own and re-crawl to update it too
It's going to change a lot the internet we knew, unless somehow we managed to agree on a common high quality dataset repository.
The status quo with those js PoW pages doesn't really benefit the server owner at all, it's wasted energy.
I mean proof of work is always wasted energy, but I figure it's better to kill two birds with one stone.
CoinHive was one example of this. (I think this is a correct link? https://github.com/cazala/coin-hive). Although I think ideally you would want to have some sort of browser plugin or app that runs on bare metal instead of a proof-of-work in the browser, because RandomX is designed such that it's slow when implemented in JS (https://github.com/tevador/RandomX/blob/master/doc/design.md)
Another option I was thinking of would be a pretty inflationary (or demurrage) cryptocurrency in which you have some sort of RandomX or other CPU-bound PoW. A web server could act as a mining pool and use mining shares interchangibly with micropayments.
You could do this mining-share method with Monero right now, it's just that you have higher transaction size in Monero and no real analogue to Bitcoin's LN-based microtransactions. Also you would want the cryptocurrency to be more inflationary (or demurrage-based) to promote usage.
Monero's FCMP++ lays some groundwork for payment channels, but it still lacks the nessisary timelocks. Also there was DLSAG which could have enabled payment channels I think, but it's no longer relevant. I also insist that you would need to change the tokenomics to favor greater inflation (maybe you could make coinbase scale linearly with hashrate?), otherwise the miner reward would be economicially insufficient.
Re audience, the audience for Gentoo bugzilla went just to zero. If even 1% of the current bugzilla users would use the micropayments, that's literally infinitely more organic human visitors than today.
If it is the latter, then, perhaps, some p2p (torrent-like) content delivery system could help.
Genuine question, I'm not up to date on how Cloudflare operates right now
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
What's the point of this compared to letting cloudflare handle everything automagically? Presumably whatever heuristics they come up with are going to be better than you can, given limited time and budget?
Scraper "attacks" don't take down our robot-specific server very often; it's safe for us to take heavy-handed approaches that sometimes redirect users there. 99% (made-up high number) of the time, the misdirected users don't realize anything is amiss.
Start by analyzing your traffic, specifically user agents. Look for "robot" or even "bot" in the user agent and load balance those to a robot-specific server. This can all be done within Cloudflare. The only code is the user agent condition. Note: I'm very open to input here if anyone reading notices that we're shooting ourselves in the feet. Based on our analysis, the remaining traffic is a good picture of our human users.
We have loads of other conditions, mostly balancing specific IP ranges for entities when we know exactly who they are, but this is a good start.
Did you miss a "dis" in there?
You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access.
Or micropayments, of course, but hard to get normal users to sign up for micropayments. Micropayments could of course be the way to bypass the crypto mining gate.
But in the case of anubis it's not even used for crypto. It's just wasted.
>You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
No, the problem is time wasted. I don't care about the electricity cost. Spending 10s to solve a challenge on a 5W SoC translates to 0.0005 cents (yes, cents, not dollars). Meanwhile if I click on a link and it doesn't load in 5s, I'm seriously questioning the value of your blog or whatever, and will probably just close the tab.
This is not only unfair to legitimate users having to pay.
It also needs to explain what actually happens when the Ddos succeeds (I know, you are talking about scrapers, but what’s the difference really?). Does that mean that the attacker just gets to shrug it off.
Don’t take my statements as facts, I just want to outline a few reason I could come up with that show a purely technical solution might not be enough.
Akin to “just use Cloudflare, it’s free”.
Yes it works but why conceding defeat and say “oh from now on you depend on a business to publish a server”?
Though, checking, I see that it requires a login but is still up. I misunderstood what 'closed' meant. This seems fine.
This is not a tech problem. This is about what should or should not be legal.
Nor it’s a question of having time to implement solution X or Y.
If someone attacks you yes, you should have better security but you also need to have legal recourse, or it will never stop.
Ddos is already illegal.
I am not a lawyer so don’t ask me for exact resources, which vary by country anyway, but stop treating scrapers as an inescapable force of nature.
If you create a law that says you have to honour robots.txt files, what do you do if an IP from another country fails to do so?
I get what you're asking, and I'm wondering the same. Not all sources are created equally and we see the results all the time. LLMs outputs nonsense all the time, like Flock cameras containing 5 grams of gold and ounces of copper, because they are completely on critical of their sources. Perhaps there's some weights that says: Kernel mailing list, MariaDB documentation and Microsofts Learning sites are 100% trust, Reddit 50%, 4Chan 10%, but I doubt it.
Anthropic might care a little bit, seeing as they scan books, but again, is it just all books? Because other than some flowery language I don't really see the point in scanning a 1970s paperback only spy novel.
Also: where exactly are AI companies incentivized to be anything but shitty 'neighbors'?
Spam, DDoS attacks and other network abuse used to cause your hosting company or provider to call you and tell you to knock it off or you'll get disconnected, if your provider was reputable. If your provider wasn't reputable, it was likely a matter of time before they would get a nasty call from their upstream provider.
Now it just gets you a thank-you from the sales team for all the bandwidth you bought.
Meanwhile, do any of the cloud providers have any incentive to do anything about this? Hell no. They're making money off you having to ramp up extra or bigger instances. They're making money off the bandwidth. They're making money off the people doing the crawling, too. They're incentivized to do the exact opposite of effectively help you with your AI bot problem.
That's it, I guess?
However, it is also easy to get large numbers of v6 addresses cheaply.
Stablecoins have a great use-case and you get paid for bots to access the site with the humans living in peace without the site getting botted.
Job done.
A headline would be great if those AI companies would close down. I hold them all responsible for this.
Then you've got good old cloudflare which is free to use
2. Considering the spike of complaints over the past two years and the fact that they range from small time site operators to larger orgs (Linux Kernel, GNOME, Duke University, etc), do you think that maybe you're missing part of the problem space?