Perhaps realizing all of this, Google hasn’t yet deprecated 2.5, bit limits access to it to “those who have used it before.”
It’s really really good for low cost search!
I am currently using Perplexity fast search and fetch, and I am happy with that. I would try our Ceramic.ai, but I need to be able to fetch the pages as well (I do not want summaries).
Don't give them (G) ideas.
Google Search only works as a business because human eyeballs see (and brains choose to click on) ads at rates that justify advertisers’ (massive in aggregate) dollars.
(Was your comment a joke? Or did google announce this through different channels?)
- Company has great initial product
- Company gets popular
- Shareholders demand infinite growth
- Company becomes rent-seeker
- GOTO 10
It means "hi spending approver, I'm going to add $100 to our CF account" instead of "hi accounting+management+security, please initiate the process of evaluating new third party vendor Foo for use in my project, I hope we can get it approved and integrated into SSO sometime next month".
Instead I put my API keys to cloudflare, set limits, and gave the agent the CloudFlare token, and in minutes it could contact tens of services.
edit: not to mention instead of loading balance to each service I could just keep balance on cloudflare that covers them all
Put it this way: I'd rather Cloudflare owns the Internet than Google, Meta, Amazon or Alibaba.
Cloudflare is however known to deliberately screw over some of their clients.
> I'd rather Cloudflare owns the Internet
I'd rather no one does.
Why would I use bonsai? Why not use ElasticSearch directly?
Given the number of people on HN who report massive problems from scrapers and other bots, it sounds like if Cloudflare doesn't do this, someone else will need to. I might have thought bandwidth was cheap enough now for it not to matter, but I guess the bots are costing some sites a lot of money.
As for the bots, I thought the same thing, but it is indeed a huge problem. They've brought my websites down pretty frequently recently. I tried Cloudflare but visitors complained, and I think you can't win against the bots anyway, so I've resorted to performance improvements and serving every request.
- let more traffic in, eat the compute cost
- block larger cohorts of traffic, affect many real users
- babysit the rules to get them just right, lose time doing that
In my experience the residential proxies exist but are not that common and many aren't trying as hard as they could. It's really a war of which side wants to spend more attention on the problem.
ceramic.ai - $0.25 per 1,000 requests
Exa - $7.00 per 1,000 requests
Linkup - $5.00 per 1,000 requests
Does anyone have insights on the quality differences? Web search API pricing for AI agent usecases has always felt so expensive for what it is, but I have no grounding on the economics of running a web index.
EDIT: formatting
Thanks asciimoo for https://github.com/asciimoo/hister
Then shortened the query to just "qwen-3.8 flash next" ... results came.. all unrelated. In fact, these were almost all paper links .... no relation to actual search term.
And I had thought that I finally had found a cheaper search alternative.
Then searched for "Cloudflare OHTTP Gateway" .. this text is literally in the title ... but zero link for this page.. the closest it yielded was this link: "https://developers.cloudflare.com/privacy-gateway/" ... it seems cloudflare updated this 2 days back.. the original content was last updated in 2022 ... so that's what the cutoff index seems to be.
There actually is such a thing as verified bots on Cloudflare that gets through most blocks (and these services are likely are part of that), but ultimately it just depends on how the website owner has things set up in Cloudflare
Verified bot is just a label. What you do with that information is entirely up to you as the operator. It doesn't say anything anything about the service and doesn't provide any guarantees about the traffic.
But does CloudFlare itself commit to zero data retention? If not, this isn’t too meaningful.
I use CloudFlare developer platform and quite happy with tools, but I didn’t use the gateway API and always used OpenRouter which does support web search.
I can see it useful for those who didn’t do any integrations or like to keep logs at one place, but did customers actually ask for this?
CloudFlare's entire business is scale, cost, and complexity. They are powering like half the web at this point. Wouldn't really call them a "new entrant".
The absence of the Perplexity Search API is to be expected though, knowing how much these two companies despise each other.