Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.
How do you figure it hurts real users? The amount of compute/energy used on the proof of work is pretty minimal. You're using more when you watch a YouTube video or browse a JS-heavy web app.
Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic.
I put it in front of my hobby project because I can't afford to serve hundreds of thousands of bot requests from residential proxies all across the world, and I didn't want to route all of my traffic through a third party company like Cloudflare.
1. It makes the web more annoying and time wasting.
2. I'm a real user and I have to enable JavaScript and run arbitrary programs from untrusted websites to access them. One browser that I use doesn't even support JavaScript (w3m). Fortunately it is unaffected because it doesn't have the string "Mozilla" in its UA and Anubis lets every user without that pass to not hurt legitimate bots like search engine crawlers. As said Anubis is trivially bypassed. The only reason it isn't bypassed is because it's simply too irrelevant to the AI companies.
I empathize, I hate the bloated JS-heavy web and avoid it when I can. I just don't see a better solution. The options for me are: sacrifice some legit users by using Anubis, or sacrifice all users by shutting down my service because I can't afford to serve the bot traffic.
There are false positives. Every time I've used a tool like this, I get reports from my users about it blocking their access. It's especially bad for users who use privacy oriented browsers like Brave or use VPNs.
For a personal site where you don't care about 0.1% of your users getting blocked, it's probably fine. For a business, blocking some of your paying customers isn't acceptable.
> For a business, blocking some of your paying customers isn't acceptable.
That depends on your business and the costs you're incurring from the bots. It might be justifiable to lose 0.1% of your customers if it means saving server costs from serving useless bot traffic.
I suppose it would depend, yeah. I find having paying customers being blocked to be completely unacceptable. Not only is a bad customer experience, but they tend to be very vocal about it. Posting online, leaving bad reviews, etc.
You are quite incorrect. I make a living off a website and 99.9% of my traffic is from bots. Literally hundreds of millions of requests from bots. The cost of bots is worth the cost of not harming legit users.
Over a slow mobile connection Anubis doesn't load at all, even if the actual website would load just fine in seconds. Hence the user is locked out from the website.
Most of the friction is just JS overhead for the computations, a compiled solver is like 1000x faster. If Anubis ever gets popular enough that scrapers care, it would be trivial to defeat. And last I checked you could bypass it by just modifying the user agent
I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.
I guess the disconnect here is a bunch of HN'ers believing professional companies and websites want to attach their branding to a sexualized anime character and that they are willing to pay to remove it.
Which one then wonders why they would install it in the first place.
In any case, if you run a professional website, this immediately comes off as juvenile and/or amateurish. And am y people just assume it's part of your website.
Perhaps you should check if anybody else sees what you see. After browsing through every image I could find in the history of this project, I, at least, am convinced: any "sexualization" _you_ see is entirely in your head.
> ... in judeo christian societies
I really hope you're not trying to whitewash your own bias by alluding to those religions. From everything I've read of their texts, and interacted with practioners of those religions, I'm unaware of any bias that classifies _cartoon depictions of children_ as "sexual".
Perhaps you should try whitelabeling instead. Don't like the packaging? Package it yourself!
I'm not talking about furries I'm talking about the mascot of the Anubis software program, which is not in any way sexualized.
Do you believe any anthropomorphic cartoon animal must be implicitly sexual because "furries exist and have sex?" I hate to break it to you but just about everyone has sex.
Also, Looney Tunes must horrify you. Wile E. Coyote isn't even wearing any clothes at all, and they show that to children!
> is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
Most people either don’t know or don’t care about “those subcultures”. I bet a lot of older people think it’s a cartoon figure of Betty Boop (nurse) and miss the furry bit since it appears and disappears quickly. Most people also don’t have a brand.
So it seems like you’re describing a concern that only affects a tiny fraction of people:
- Not interested in paying for custom branding, so obviously not a corporation or influencer
- Dislikes cartoons
- Aware of, and hostile towards, “furry” subculture
That has to be an exceedingly small fraction of potential users of Anubis, and given how much businesses and branders will pay to custom-brand something, I’d counsel them to stay the course. Sure, a few never-payers will never pay, but they wouldn’t have anyways, so they can cope with Nurse Betty or look elsewhere for a competing free product.
If you think about this in physical market square terms — in other words, a bazaar — it seems horrendously rude to complain about a shop logo sticker on a free product handed out to anyone that walks up and asks for it. If you want it white-labeled so you can write your own name/logo on it, you pay for the privilege of displacing their name with yours. But you don’t stand there and loudly complain that their shop mascot has dog ears while holding a freebie bag of product, without losing the respect of everyone who hears you doing so.
I'd argue its rather functioning exactly as intended with regards to obtaining paid users.
Also this view seems a bit elderly. For most towards the end of the millennial curve and younger, anime is no longer subculture, its just general culture at this point. Although I would agree it isnt necessarily what you'd want for every platform and web presence.
How hard is it to maintain a fork that changes nothing except the logo?
If you can't be bothered to maintain a trivial fork, then why should the author of anubis be bothered to serve your branding needs? It's not like you have a service contract or anything do you?
It's MIT licensed, you are free to do whatever you want to it, including removing the logo. They're basically just saying "we'd prefer you didn't do this, but we're not preventing you from doing it."
You wish they'd give your preferable imagery for free and not make you feel bad for using their free software for personal gain. Your subculture is irrelevant and not special.
It seems to come up every time this complaint arises. "I want to use an open source project without being associated with furries" isn't that unreasonable of a request.
There is a difference between some maintainer somewhere being involved in a subculture and graphics from it being the first thing you see when you visit an unrelated website.
My problem is more that I don't want open source infrastructure I don't control to be infected by this. For my own website I can just make it efficient enough not to need hacks like this.
Nah, I don't especially like the logo myself when I come across those sites, but I do like that "brand" people who want to make money off it and not pay dislike it even more.
It's really smart of them to manufacture a negative sentiment about the project because the sexual fetish icon gets injected into all kinds of places where it is absolutely inappropriate?
Someone willing to take the 2 minutes to ask Claude to change the logo can probably also ask it to fix any bugs they find, or add any new features they might want.
This is like getting angry at cookie banners instead of all the companies tracking and selling your data.
You're complaining about the symptom (needing to have these checks) not the cause (if they don't, 99% of their traffic will be bots, the site will slow to a crawl and be unusable anyway).
In any case, I saw the dumb anime girl for about 2s then the site loaded. Not a big deal.
The GCC bug tracker uses the meta-refresh challenge, which does not require JavaScript. Due to the fact that the server makes sure the client has waited at least 75% as long as it should, the HTML has to add one second to the meta-refresh wait. Patches welcome. Meta refresh granularity is in single digit seconds.
I'm assuming a bot running a headless browser instance can still get past it?
It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.
People tried this in 2013 or so, there's no point to it. Doing proof of work in javascript in a browser is so crushingly, pointlessly slow that there's no value at all. Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.
To help you combat these unwelcome scripts, we’ve partnered with Disconnect and compiled lists of domains that serve cryptomining scripts so you can block them directly through Firefox. When you download Firefox, tracking protection is enabled by default, blocking known cryptominers.
Technically true, but the list is mostly nonfunctional - cryptominers are basically extinct in the wild, and most of the domains on the list have expired or no longer serve mining scripts.
Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, it's security by obscurity being dressed up as something more.
pow_server http://127.0.0.1:8080 backend avx512-x16
──────────────────────────────────────────────────────────
uptime 00:03:12
solver ● BUSY difficulty 9, 0.3s
queue [####################............] 5/8 peak 12
──────────────────────────────────────────────────────────
accepted 1240 solved 1180
503 shed 48 504 timeout 2 4xx/5xx 10
──────────────────────────────────────────────────────────
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
hashes 3.90GH total avg 65.3MH/s
Ctrl-C to stop
Claude even made a nice little API server for it after implementing midstate compression, AVX multi way hashing, and a CUDA kernel. This doesn't stop the literal LLM it's trying to block from solving the challenges, it's really annoying that everybody is using it and claiming that it's something that's usable in the real world as a result of it using proof of work. It's obscure, and obscure is fine so long as nobody is pretending that it is secure.
It is not absolutely useless, empirically, which you'd discover if you had a website getting hammered by bots and experimented with anubis as a countermeasure.
While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.
Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.
Targeted attacks may not be repelled at all. That's not the point.
It does not even require the PoW thing Anubis does. I've setup a simple logic that just:
Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.
Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots.
Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op
I've extended this with a FCRDNS checked exclusion for Googlebot.
Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.
Also, I only apply these rules to routes that are not cheap and cached.
That's not far from what anubis does for clients that are determined to have light souls. It doesn't always send a PoW challenge.
For a webapp that sets a long-lived cookie, that cookie could be used to bypass anubis completely, or lower the weight in anubis so that it doesn't send its pow challenge unless there are major red flags. If bots start to abuse that exception, it can be removed.
I once discover you can put escaped XML or json in css content. The purpose was to have static data sets that work cross domain. No headers to configure no letting strangers run all you can eat malicious js on your site.
Define "scale". For any reasonable wait that you're willing to impose on your users, any PoW scheme heavily favors attackers. They have unlimited time and can be scraping even while they're asleep. Your visitors on the other hand don't have that luxury. You might argue that's not the point and it's only to stop dumb scrapers that are effectively ddosing your site, but if it's just dumb scrapers, you could've stopped them less onerous measures like tls or javascript fingerprinting.
It's more or less designed for it, it's SHA256 with a break in the middle for midstate compression to be effective, and the difficulty system is based on a misunderstanding of how bitcoin PoW works ("number of zeros" is never, ever a consideration in bitcoin, it's a match to a floating point target).
sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.
write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to be met is in this repository. https://github.com/TecharoHQ/anubis
Then
let’s develop this more. turn this solver into a local HTTP server that can be given work in the request, and it returns solved work. make an end to end tester that sends test work to the solver and waits for a valid response. add support for solving with a GPU using cuda.
Then it was done more or less, it happily made a local server that supports solving the challenges given to it in bulk with priority based queue and can tolerate potentially tens of thousands of requests a second with no issue. The CPU time spent solving the challenges is less than the SSL setup for the connections. The GPU version does in excess of 20GH/s (but with high latency) though I didn't really test it, I'm not using this for anything but proving a point that the LLM itself can write the bypass tools and run them happily.
I'm almost ready to ship the wasm feature in the next version of Anubis after the one that's about to come out.
The big blocker is that testing against dozens of googles chrome to ensure functionality on abandoned smart TV oses takes a nontrivial amount of time. As an example of the level of debugging and the like required: https://github.com/TecharoHQ/anubis/pull/1684/changes/67621f...
My office gets very warm when chromesweep runs.
This is something that is complicated enough that even though LLM tools can help, it's not a magic bullet. It's just complicated in general.
[1] https://anubis.techaro.lol/