I'm probably just being naive though...
You can of course argue a lot of edge cases if you really want. For the most part I want to say "it isn't worth the argument". In some cases I will take your side if I really have to think about it, but in general the system google has been using mostly works and is mostly an acceptable compromise.
Also most people would agree they are fine with being indexed in general. That is different from email spam where people don't want it.
People are generally fine with indexing operations so long as you don't use too much bandwidth.
Using AI to summarize content is still and open question - I wouldn't be surprised if this develops to some form of "you can index but not summarize", but only time will tell.
Along with all the other AI companies out there, the've committed the biggest theft in human history.
Then they bend over backwards and do the "but not like that!" crap with their legal team and swing their wealth and influence around to screw over other companies and people, and a vast majority of it just vanishes, gets memory holed, with NDAs and out of court settlements, so you never get to see the full scope of harm they inflict unless you're watching like a hawk and catch the headlines before they get disappeared.
Google needs to be broken up and we need to legislate the dismantling of the current adtech regime, with a privacy and sovereignty respecting digital bill of rights that puts the interests of individual citizens above that of giant corporate blobs and the mass surveillance data industry.
> SerpApi’s answer to SearchGuard is to mask the hundreds of millions of automated queries it is sending to Google each day to make them appear as if they are coming from human users. SerpApi’s founder recently described the process as “creating fake browsers using a multitude of IP addresses that Google sees as normal users.”
this has to be satire. Is Google not the #1 entity guilty of exactly this?
They abuse this power to scrape your work, summarize it and cut you out as much as possible. Pure value extraction of others' work without equal return. Now intensified with AI
But yeah, you're right. They're not deceptive
nobody is forcing anyone. This is the same argument that people said about google search. Nobody is forcing anyone to use google search, google chrome, or even allow googlebot for scraping.
Thousands of poeple have switched over to chatgpt, brave/firefox ..
Your argument sounds like "I dont like Apple's practices, and I'm forced to buy iPhones. No buddy, if you dont like Apple, dont buy their products"
If you want people to visit your website, limiting yourself to the "thousands" of people who don't use google isn't really an option.
> Your argument sounds like "I dont like Apple's practices, and I'm forced to buy iPhones. No buddy, if you dont like Apple, dont buy their products"
Well, I don't like Apple's or Google's practices, but I basically [1] have to use either iOS or Android.
[1]: yes there are things like GrapheneOS and librem, but those aren't really practical for most people.
No, not really. There are alternatives to Apple. Whereas here Google controls the gate to the majority of internet traffic
For many it's "block Google and your business dies"
They have a different definition of "licensing" than most people I guess. Aren't site operators complaining about Google using this "licensed" content in AI overviews... not to mention the scraping for AI model training.
The pot is calling the kettle black.
SerpApi doesn't have that privilege.
https://radar.cloudflare.com/ai-insights#ai-user-agents-foun...
DDoS remains illegal regardless of robots.txt.
> Stealthy scrapers like SerpApi override those directives and give sites no choice at all. SerpApi uses shady back doors — like cloaking themselves, bombarding websites with massive networks of bots and giving their crawlers fake and constantly changing names — circumventing our security measures to take websites’ content wholesale. [...] SerpApi deceptively takes content that Google licenses from others (like images that appear in Knowledge Panels, real-time data in Search features and much more), and then resells it for a fee. In doing so, it willfully disregards the rights and directives of websites and providers whose content appears in Search.
To me this seems... interesting, for sure. I think that Google already set a bad precedent by pulling content from the web directly into its results, and an even worse one by paying websites with user-generated content for said content (while those sites didn't pay the users that actually made the user-generated content, as an additional bitchslap.)
But it seems like at the very least Google is suggesting that SerpApi is effectively trying to "steal" the work Google did, rather than do the same work themselves. Though I wonder if this is really Google pulling up the ladder behind them a bit, given how privileged of a position they are in with regards to web scraping.
It's a tough case. I think that something does need to ultimately be done about "malicious" web scraping that ignores robots.txt, but traditionally that sort of thing did not violate any laws, and I feel somewhat skeptical that it will be found to violate the law today. I mean, didn't LinkedIn try this same thing?
Like GoogleBot?
And yeah, robots.txt is not enforced by any law.
I think this is just about dragging SerpApi through a lengthy legal procedure and fees.
Reddit Accuses 'Data Scraper' Companies of Stealing Its Information
https://news.ycombinator.com/item?id=45695433
Our Response to Reddit, Inc. vs. SerpApi, LLC: Defending the First Amendment
And then pretending that they're fighting for other people's copyright is just the cherry on top of the pile of hypocrisy.