Love this. There’s been past discussion on HN re how OSS maintainers are treated, and this is such a wry sentence. I really appreciate the tone / attitude to the problem.
So this is going to be interesting.
My plea (in a small voice) to the OP: just as many sites still do for JavaScript, please add a "This captcha required Webassembly to continue" message to your code when Webassembly is disabled. And, incidentally, for smaller platforms/browsers that don't offer a Webassembly engine at all.
> One of the big things that blocked this shipping for so long was not having an escape hatch of some kind to allow clients that disable WebAssembly by policy to get through the gate. [...] This combination of factors means that there would need to be another implementation of the proof of work code in JavaScript that would actually execute the number crunching.
This amounts to giving Google effectively complete control over the web. I'm not sure that's a good idea. IMO it's important that Firefox and Safari remain popular enough that webdevs are forced to support them.
I'm somewhat sympathetic to "If you want to 'customize' your web experience, you shouldn't be surprised when things don't work right." in terms of turning off commonly-used features, but the request isn't to make it work right. OP just wants a helpful error message.
I have a Mac from 2014 running Yosemite that I occasionally use to test for backwards compatibility in my own frontend code (for fun!). But IMO the best way to ensure compatibility is to use period-correct toolchains or toolchains where the pace of change is slower, like ClojureScript.
I assume that the previous challenges will still be available for circumstances where WASM is not available (unless we are just working on the basis that it is available if JS is?) or perhaps as a fallback. TFA explicitly mentions “smart” TVs, and I'm sure there are plenty of those out there that run browsers old enough to not know what WASM is. TBH these would not be a priority for me, but they obviously are for this project given they (well, TVs generally, not specifically elderly ones) get specific mention in this write-up.
If so, what is to stop scrapers from just forcing the fallback POW option instead of trying to run the new memory-hard algorithm at all?
On https://wasm-feature-detect.surma.technology it shows that I don't have 3 of all these features but I'm not sure if Anubis needs any of them to not kick me back to the pure JS solution
Which would apparently be bad because
> The WebAssembly that's shipped with this flow is ridiculously performant. This may mean you need to adjust the difficulty [upward, to avoid that bots solve it trivially when they support wasm, I assume this means]
combined with
> The wasm2js flow doesn't currently have a way to update the progress bar [so you have no clue about remaining time]
(wouldn't be the first time that I gave up on a page because it was stuck on 0 hashes per second)
I feel like Anubis is ironically speedrunning a lot of discoveries the crypto folks have already made several years ago...
And while I’m sympathetic to the idea of not wanting to run JS, to a first approximation modern browsers are JS engines that have graphical displays. How things should be vs how they are is a classic is/ought problem. The world took a vote on what a browser’s meant for and we lost. Fighting it today is rough; tomorrow, futile.
No, because there are technologies that don't have this issue, eg. privacy pass.
We could link it to a site, you generate for HN, I for Reddit, but it so happens that you visited Reddit more and I HN, so we depleted our Anubis POW, so we could exchange some Reddit Anubis with some HN Anubis.
Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.
I don't understand the units here, wouldn't a bit versus nibble difference make the multiplier 16, instead of 1024?
The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it.
So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
As you said, if you actually want to interact with a remote git repo you'd do so with packfiles not diffs.
I've just finished creating a WASM 1.0 implementation (simple JIT on x86 and x86_64) for plugin usage and I'm looking on how to support more languages beyond C (for which I have "micro-libc" to create really small binaries). In particular C++ and Rust so the article contains useful info for that. C++ that would be more problematic I think, would have to resort to full Emscripten or something... I've tried to compile standalone C++ stdlib but wasn't much sucessful yet.
BTW, I see in the screenshots you also test Firefox, I think it would make sense to test the latest two ESR versions (these are often used by Linux distributions).
- https://techaro.lol/robots.txt
- https://anubis.techaro.lol/robots.txt
and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
Scrapers can't change their strategy? It's not hard to think of mitigations: keep on using a session until you're banned, and failing that rotate your IP. Nowadays you don't even need to think. You can probably tell a LLM "write me a scraper that has adaptive anti-anti-bot mechanism".
Yes, but you normally don't try to fight all strategies up front, you add stuff on a need-to basis. Currently, lots of people are in the stage of "need to get this thing using bunch of IPs to stop hammering my servers", apparently Anubis helps them with that. If that stops working, other tactics are tried, and so it goes on for the life of your service. Hasn't stopped yet, doesn't seem to be stopped soon either.
Turns out that in practice there's some bots that implement support for this but not JavaScript.
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
"Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.
And then regular users: who now can't really browse your site.
And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down.
The solution basically makes the favored customer well funded AI scrapers, not users.
That's entirely up to how the site operator configures the PoW challenge.
> anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down
Do you have any evidence that it is the large well funded AI companies DoSing sites by (for example) scraping absurd numbers of permutations of cgit url parameters?
Even if it is (which I doubt), if all this accomplishes is getting them to spend 10 minutes optimizing their bots to not do that it will be a major win.
That would defeat the purpose. The goal is to make scraping costly, not profitable.
But, in this world, the website owner would receive tokens that they can then use to do whatever they want, including paying for servers and bandwidth. This is the sense in which the cycles aren’t wasted: the website owner now has cash to spend.
Effectively, both scrapers and ordinary users would be paying for the privilege of getting website bytes.
This also solves the problem of having to wait for your phone to solve the challenge while you’re browsing: you can buy or mine some tokens ahead of time and pay them as soon as challenged. So can the scrapers, but because they’re accessing enormous numbers of pages it’s hopefully prohibitively expensive for them.
> This makes Anubis challenges use a memory-hard proof of work function (argon2id) instead of just a CPU hard one. It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
Nice.
Meanwhile BOINC is well-established as the platform for distributed computation. If you can figure out how to squeeze its work units into Wasm challenges, I'm sure a lot of researchers would thank you.
…<img not found/>…
I don’t know what this picture was supposed to be, but a 404 demonstrates your point perfectly well.
https://github.com/eternal-flame-AD/pow-buster - Browser extension that already used WASM to accelerate the anubis solver, among others.
https://github.com/kasper93/anubis_webgpu - Browser extension that uses webgpu, for another order-of-magnitude speedup (depending on your GPU of course).
Lmao yeah no. I don't think a little argon2 is going to change shit all.
I mean the thesis of Anubis itself is "scrappers are compute limited (in ways that consumer devices are not)" which has its own massive flaws.
You sound like a child throwing a tantrum. Before throwing around insults calling people post-teenager, you should maybe look at your own behavior. Is adult behavior getting angry to the point of swearing because you saw a cartoon character?