Hacker News new | past | comments | ask | show | jobs | submit login
Tell HN: Impassable Cloudflare challenges are ruining my browsing experience
523 points by blakeashleyjr 17 days ago | hide | past | favorite | 304 comments
I travel often. Sometimes I use a VPN, sometimes I don't. I use a heavily customized Firefox config on Linux.

Cloudflare challenges have made large portions of the web unusable for me.

Some recent examples

  - The "unsubscribe" button in Indeed's job notification emails leads me to an impassable Cloudflare challenge. The "Contact Us" page is also behind an impassable Cloudflare challenge.
  - While migrating a non-profit off of A2 Hosting, their login forces me to re-enter credentials after failing a challenge, looping endlessly.
  - On a particularly ironic note, I tried to complain on the Cloudflare Forums—met with another impassable challenge.
When reachable, customer support always says "try a mobile data connection", "switch to Chrome", or some other variant of "too bad, so sad".

Is anyone else dealing with this mess?




> The "unsubscribe" button in Indeed's job notification emails leads me to an impassable Cloudflare challenge.

That's a CAN-SPAM act violation.

FTC: "Tell recipients how to opt out of receiving future marketing email from you. Your message must include a clear and conspicuous explanation of how the recipient can opt out of getting marketing email from you in the future. Craft the notice in a way that’s easy for an ordinary person to recognize, read, and understand. Creative use of type size, color, and location can improve clarity. Give a return email address or another easy Internet-based way to allow people to communicate their choice to you. You may create a menu to allow a recipient to opt out of certain types of messages, but you must include the option to stop all marketing messages from you. Make sure your spam filter doesn’t block these opt-out requests."[1]

Experian was recently fined for making it hard to opt out of their marketing emails.

The actual regulation text:

§ 316.5 Prohibition on charging a fee or imposing other requirements on recipients who wish to opt out.

Neither a sender nor any person acting on behalf of a sender may require that any recipient pay any fee, provide any information other than the recipient's electronic mail address and opt-out preferences, or take any other steps except sending a reply electronic mail message or visiting a single Internet Web page, in order to:

(a) Use a return electronic mail address or other Internet-based mechanism, required by 15 U.S.C. 7704(a)(3), to submit a request not to receive future commercial electronic mail messages from a sender; or

(b) Have such a request honored as required by 15 U.S.C. 7704(a)(3)(B) and (a)(4).

That seems to cover it. File a CAN-SPAM act complaint (spam@uce.gov). Send a copy to the legal department of the sender.

[1] https://www.ftc.gov/business-guidance/resources/can-spam-act...


Thanks for that note. I receive „spam“ by a US based Car Rentel/Leasing Company, cause they prevent me from unsubscribing because i am in European IP-Range (geo-blocking). Especially „nice“ cause they send me contract specific details of one of their customers, who misspelled his email address.


I'm in a similar boat. A UK bank thinks I'm one of their customers (someone with a similar name). The reply address is no-reply@ and I'm not about to call a foreign bank.


I had the same happen with a AU insurance company that also made it hard to reach them.

I sent an email to their regulator that this company keeps sending me confidential information about one of their clients. It took one day until I received an email from the company informing me that they've corrected the mistake and I shall no longer receive any emails, and it worked, I haven't received a single one since.


Maybe I’m lazy but why do above four posters do so much effort?

I just mark as spam and or block the sender


If I made a mistake while entering data, I'd be happy if someone told me they receive emails from me that they probably shouldn't be getting, so I do the same when it's not obvious spam/scam.


same. although, if it's a reservation you're being sent, you can cancel it to let the person know they're using the wrong email (plausible deniability because you don't recognize it, yet are getting a reservation)


How would the person know their reservation is cancelled?

reply


they'd show up for the reservation they made and find that it doesn't exist!

reply


Quick note that if you use a proper email hosting service, or host yourself, you can add a sender block rule to eliminate this nuisance.


Are there email services which don't allow you to block addresses or keywords?


From: no-reply@ and simlar fake senders should just result in immediate rejection of the mail at submission time.

Tempted to set that up on my server.

reply


Getting a U.K. bank account without having a U.K. mailing address isn't the easiest thing in the world to do. Maybe someone would be interested in acquiring it from you.


Is this Hertz? Somehow Hertz in Mexico decided to add my email address to their mailing list, and I tried to complain to every level of Hertz to get them to stop. Their hosters didn't care, their upstream didn't care.

I decided to download larger files from their web site a few tens of millions of times, which I think cost them a few hundred dollars. Unethical? Perhaps, but I'm not the kind of person who just accepts that companies are too large to have humans that can communicate and that I should just accept their harassment.

It worked, though. I finally got a response from Hertz saying they were going to "get to the bottom of it", and I finally stopped getting their spam.


When you say "it worked" referring to you downloading big files to generate cost for Hertz, it mean you told them you were doing this and would stop if they remove you from the mailing list?


I received spam for quite a while from Robinhood, back when they suggested they were going to enter the UK market and to sign up for more details.

They didn't but I still recieved spam which I couldn't opt out of because they wanted me to log into my account, even for support, which obviously didn't exist.

At least back then we had Twitter and messaging them publicly got a customer service response.


I know that feeling all too well. There's an Australian guy with a very similar email address that keeps entering it incorrectly, and I end up with the promo emails for these accounts. And because some of them are geolocked to Australian IPs, it's impossible to unsubscribe via the links in the footer.


I get emails all the time for some person in the US who must misspell his own email address. So far I’ve cancelled his haircut and car garage booking.


Unfortunately the government seems to have given up on enforcing the CAN-SPAM act. If they actually enforced it spam companies like Salesforce would face massive fines.


You can press charges yourself and get lawyer fees for your efforts. Probably not worth it, but you don't need the government to do this.


Individuals cannot "press charges", nor can the police. Only a state/federal government attorney can file charges against someone.

A person or police officer might recommend some action to a DA, but it's completely up to their discretion what to do with that information.


Yeah, the same way I don't need to pay taxes...


I would suspect that OP is choosing the webpage out of convenience but that there is a List-Unsubscribe: header hiding in the raw version of the email, cheerfully nuking the FTC complaint. Now, demonstrating that the List-Unsubscribe worked is left as an exercise to the reader, but let's be honest, it's the same with the web page variant with bonus points for those pages usually ending it "yeah, we'll get around to it is 364 business days" or some shit


Makes me wonder. If it's illegal to deploy bot protection on unsubscribe links, and there are massive lists of leaked email addresses available, surely someone has tried to mass-unsubscribe tons of email addresses from tons of mailing lists?


Unsubscribe links can include additional authentication information, as long as it is all included with the link in the mail.

If you don't do that, bot protection isn't going to stop a dedicated troll.

reply


More likely: someone tried to automate unsubscribing one e-mail address from massive list of mailing lists.


brb unsubscribing the world from Indeed


Don't worry, the next administration is likely to eliminate any rules like that.


Inane low effort comment.

CAN-SPAM was introduced by Republicans and signed into law by Bush btw.

reply


It feels weird that a completely US based Recruit acquisition shows such a typical Recruit behavior...


"Visiting a single Internet Web page" is considerably more involved than that. In practice, it means making a request to the DNS servers and running Javascript that's injected by the CDN/proxy which "verifies" (runs some heuristics) that you're allowed to load that page.

It's like a restaurant that complies with a local food access requirement to be open at a certain time... but only by having a drive-through that requires you to not just be a human being, but also to drive a car to get to the restaurant.


You're collateral damage in the web's war against bots :(

Unfortunately, I think the Cloudflare challenges are designed to filter out users similar to your profile... once you stray far enough from the norm, it just looks like a bot / suspicious traffic to them. Statistically there's not enough users like you (privacy-conscious Linux users on nonstandard browsers) for them to really care enough to do anything about it. Site owners don't care either since you're usually like 1-2% of users at most, and typically also the same ones who block ads, etc., so they don't mind blocking you... it's sad, but I don't think there is really anything you can do about it except conform. It's an ongoing arms race and you're caught in the middle.


The sad part is that it's trivial to get around CF's bot protection if you're writing a bot (just use curl-impersonate and buy residential IPs), but it's pretty much impossible to bypass as a human if their magical black box doesn't like your browser and/or IP address.


It's the same for spam email, yet most spam gets caught in spamassassin rules that were written 20 years ago and haven't seen much improvement since then. Most bad guys just don't bother to do anything above the bare minimum. For example, I see lots of email getting caught in a rule that checks for incorrectly formatted pseudo-Outlook mailer header, which is trivial to circumvent if you pay any attention to it (the difference is in excessive whitespace, or a slightly incorrect "Outlook" version, or something like that).


see also: The surprising effectiveness of simply asking the spam server to try again(sometimes called graylisting). It shouldn't work at all, but proves to filter an awful lot of the worst mail noise.

http://man.openbsd.org/spamd


> it's pretty much impossible to bypass as a human if their magical black box doesn't like your browser and/or IP address

There are residential-IP-backed VPN services that you can use just like commercial VPN services — but they're mostly built on the backs of botnets, so it's ethically questionable to use them.


FWIW, StarVPN claims to have "ethically sourced" IPs. That is, not from botnets. Their pricing is quite a bit higher than many (cheapest plan is $20/month), but could be worth trying.

https://www.starvpn.com/


The "residential VPN" providers setup fake ISPs or buy AT&T/Verizon business circuits with large blocks of IPs and sell them as residential.

They are easily detected if you are buying IP intelligence from one of the higher quality providers: https://app.spur.us/context?q=STARVPN_PROXY


The linked page shows a sign-in screen.


Spur access requires a free account.


That's helpful to know. I wasn't aware of this.


You could also use Tailscale back to your own IP if the goal is not having to trust public WiFi.


To note, IP is only a part of it, and the full extent of what's baked into a CF score will never be explicited (for obvious reasons).

CloudFront being way past the simple blocking of IP addresses, I wouldn't be surprised if a mismatch between your IP block and your language/cookies would be enough to lower your score.


This is great for bypassing the server side bot detection but not the client side one, where it will attempt to verify the integrity of your browser environment.


Well yeah, if you’re a legitimate user, CF will block you.

It’s only easy to bypass if you’re scraping or doing nefarious stuff.


Surprisingly, it still works as intended. Yes, it won't keep professionals and dedicated bot-fabricators out, but that's like 5% of the botters out there; the rest are the bot equivalent of script kiddies who can't be bothered, and it filters them great. Meanwhile, the script kiddies have a process that still works on non-CF sites, so they don't need to improve their process.


We bypassed it by switching to starlink. Now my IP address is a too-big-to-fail CGNAT.

The old IP address was a mom-and-pop CGNAT.

Thanks CF, for protecting us from capitalism, I guess?


That's same for almost all surveillance/tracking tech. It's always trivial for criminals/abusers to bypass. The surveillance is just about controlling the sheep.


How does it get around captchas?


If they don't think you're suspicious they don't make you do the captchas, and as others have mentioned you can always outsource it to captcha farms. There are also AI models which do a fairly decent amount, and since most captchas let you repeat attempts with new patterns you can have a pretty high error rate to get past them. Then there's the ADA, which requires accessibility- many captchas have an audio component as a backup and those are easy to interpret by models.


curl-impersonate doesn't solve CAPTCHAs, but the goal is to look enough like a human that Cloudflare doesn't present a CAPTCHA in the first place.


Cloudflare turnstile isn't even a captcha. The user just has to tick a box. Behind the scenes there's a javascript challenge to make sure you're vaguely a browser and not some script a bazillion requests per minute.


It's also used for proof of work as many scrapers are using thousands of IPs but only a few CPUs


You pay contract workers in a third world country a tiny amount of money per day, to spend all day clicking boxes.


> Site owners don't care either since you're usually like 1-2% of users at most, and typically also the same ones who block ads, etc., so they don't mind blocking you

I do believe that it is true that many site owners wouldn't care. But I suspect that in the vast majority of cases they don't actually know. Cloudflare probably shows them a nice dashboard about all of these blocked "threats" and they don't know better than to question it.


While you hit the nail on the head, I am still surprised that so many tools targeted at people like me (web hosting, developer tools, etc.) are protected that way.


Its not only about protection, most web developers would use Cloudflare since its a free CDN and would increase the app load time considerably.


You can separately configure (to a large degree) the caching vs protection features, though.


increase -> decrease


Except for those encountering that dreaded captcha.


Because if such hosting and developer tools are not protected against bots, the tools end up used for phishing, spamming, etc.


I'm convinced that's mostly incompetence on the side of the companies that implement that protection.

"We have a problem with bots" - "Just create a firewall rule, whatever"


What other way would you suggest to protect a free service from bots? Cloudflare is often the easiest to implement and has a generous limit on their free plan.


Oh, they absolutely are, I don't disagree -- I use them too.

But the immediate response to bots shouldn't be "make everyone go through a captcha". There's lots of nuance that you can tune to deal with your particular situation, but the first thing I'd do is block known bots or ASNs, set up a limit to trigger (bots usually don't make 1 document request a minute), set up higher limits for users who (seem to) have a valid cookie indicating that they are logged in, set up different thresholds for certain countries that are more risky etc etc.

What you need to protect your service depends on your situation, it's not a one-size-fits-all solution. E.g. I find that I have no automated contact form spam once I add a simple JS to add some data that isn't standard, but I'm sure that wouldn't hold up if there was enough incentive to try to get past it.

But the OP mentioned not just free services, but e.g. webhosting logins. That's just sad, as is Cloudflare's community being behind an aggressive captcha. I'm a user, I'm logged in, I've posted before, I'm in good standing, yet when I go there, I need to solve a captcha. When I then go there again an hour later, guess what, another captcha.

Either there's another reason I'm not seeing or it's just lazyness as in "we need to have a forum but we really don't want to spend any resources on it, just put up an aggressive captcha that'll filter out most bots and everyone but the determined users".


Fwiw, Cloudflare does do a multivariate confidence check which is why it has multiple tiers: no captcha, a one-click captcha, the annoying puzzle captcha once, the annoying puzzle captcha six times in a row.

> I'm a user, I'm logged in, I've posted before, I'm in good standing, yet when I go there, I need to solve a captcha.

Though consider the fact that taking over someone's account shouldn't give you (a spammer) unlimited access either. The spambots you see on Twitter are mostly cred-stuffed accounts. It's a hard problem. Existing accounts are more dangerous than fresh accounts.

Imo, "write your own password" should be a thing of the past. Services should just auto-gen a password or there should be a way to require the OS (like a password manager) to generate one to avoid cred-stuffing. We're letting down the average person by making them come up with unique passwords for every service instead of just helping them. Though I'm way off topic.


> Though consider the fact that taking over someone's account shouldn't give you (a spammer) unlimited access either.

But it's not unlimited access -- it's _read_ access at that point. This is just when trying to access the forums at all, not when trying to post a message. And if they were worried about evildoers scraping all the data from their forums, they could rate-limit and then require captchas (their WAF settings make that trivial). But they don't, or the rate limiting is so generous that I've never hit it, and their forums are not that active, so I don't think that's the reason.

Adding more protection to an endpoint where users send posts makes some sense, but for reading? On their dashboard you need to solve the captcha on the login-form. On the forums, you cannot even get to the login (which works via the dashboard, where you'll solve a captcha again) until you've solved the captcha.

I use and like CF's products a lot (I'm a paying customer, I'm not even looking for free support on the forums, but their docs are lacking a lot of information that I'm interested in), so I don't believe in "we're incompetent", keeping the resource-investment low by filtering out bots and a chunk of users makes a lot more sense.


> Fwiw, Cloudflare does do a multivariate confidence check which is why it has multiple tiers: no captcha, a one-click captcha, the annoying puzzle captcha once, the annoying puzzle captcha six times in a row.

That's not correct, Cloudflare challenge pages / Turnstile will never show you a puzzle.


Most developers I've met were actually similarly lazy... we just use Chrome on Mac, and don't really want to deal with VPNs unless our employers force us to. The last few Firefox holdouts also switched after running into various WebGL/Canvas/etc issues. The same attitude that leads us to focus on "happy path" users and ignore edge cases often also causes us to sheeple into that same basic dev group. Long gone are the days where most devs custom build Linux boxen from scratch and compile custom kernels to our liking...

Anyway, I know the "Cloudflare's monopoly gating is killing web openness!" meme is common online, especially on HN, but in real life I've never actually heard anyone else complain about it (either a fellow dev or a customer or a manager). Instead, it's been universal praise for the actual issues Cloudflare exists to solve (CDN, bot protection, serverless, etc)... they are a godsend for small businesses that otherwise get immediately flooded by spam requests, especially from China, Russia, and India.

And if you think Cloudflare is bad, it was even worse before they became dominant, with terrible services like Incapsula/Imperva charging way more but providing both worse bot protection AND more false positives, or the really hard early reCAPTCHAs (that Cloudflare was largely able to replace, for users who DO fit within the "norm"). That, or you'd have to fight every random sysadmin with their own lazy rules, like firewall rules that blacklisted entire regional ISPs and took weeks or months to resolve, if they ever even checked their emails.

As inconvenient as Cloudflare is for users who take privacy seriously and try to be less trackable, for the other 90% of us who don't care as much and easily fit into their "norm" model, it's much nicer than what came before. Site downtime and slowness are also much less common now, in no small part because of their easy CDN and caching.

From the implementation side, I've set up a few Cloudflare accounts in my career, but do take the time to try to configure it to balance security vs accessibility for any given target audience. Sometimes we'd block entire countries, other times we'd minimize security to ensure maximum reach, but usually we'd customize rulesets in the middle for any given company & audience. I never got a complaint about it (our emails were still available and not blocked).

This was always a direct response to some business need, usually spambots or DDoS attempts that fail2ban etc. couldn't catch well enough. For the business, it was usually a "shit, our website is down again, what is it this time", and the choice between "for free or $20 we can get it back up again and not have this issue anymore" or "we can spend thousands of dollars and weeks of labor building our own security solution" is pretty easy. "What about that one guy who is proxied behind TOR and three VPNs with a random user agent using a text-only browser he wrote himself?" never really factors into that process =/ There's just not enough users like that out in the wild vs the very real constant threat of bots and malware.

It's a shitty situation that the web is like this today, and I wish it weren't the case, but it really is an arms race, and these imperfect weapons are just what most of us have access to...


That is not an excuse to give in to the cloudflare's agenda of centralizing everything. Bad things have happened, is happening and will continue to happen if one entity has this much control over the internet traffic


> spam requests, especially from China, Russia, and India.

On my small website, bot traffic is almost entirely from DigitalOcean VPSs.


>developer chrome, Mac

Maybe in your country, but tons of countries outside of the US (first world) avoid Macs like the plague and just use Linux/Windows as building machines.

But you are right on Google/Cloudflare, they are the poison of the web.


They are not targeting people like you. Bots are the target. If you look like a bot, how are they going to distinguish?


Their problem. They are not entitled to make it other people's problem.


They solved their problem. No matter how upset you are about it, the rest of the matter is your problem.


Though in this case it is legally their problem as unsubscribe links are protected by law in the US.


If I have a process that works for 95% of the people, why should I care about outliers who use Linux behind a VPN on a heavily customized version of Firefox?


Maybe you should try to care about something other than just your bottom line. I'm sorry if this sounds mean, but this attitude just turns the web into a giant monoculture because you can't be bothered to care. It actually ends up hurting everybody in the long run. Look how long we were trapped with IE6. Amazing how people forget history so quickly.


Everyone has limited resources. As a for profit company, the focus has to be on your bottom line. How many resources should a company use for some obscure corner case when the user can make changes?

Of course accessibility is important - ie screen reader compatibility.

A typical testing matrix in the US would be

- Safari for iOS

- Chrome for desktop and Android

- maybe Safari for desktop or you just tell Mac users to use Chrome

- Firefox if you have the time. But if not, no big deal.

We are definitely not going to test for a highly customized Firefox on Linux running over a VPN.


There’s no test to be done there. Just respect web standards and do geoblock if you want to.

The issues I have are website pretending to be apps and apps that are SPAs for no reasons.

reply


> Maybe you should try to care about something other than just your bottom line.

You can do so when your bottom line is healthy. Otherwise you go out of business. That’s business 101.


By that logic, why care about accomodating anyone with a disability? Your site works for 95% of people, why care about those who need to use screen readers?

And before you say "that's their choice," you're the one who is breaking the functionality. Nothing about using a VPN or linux or Firefox creates any problem for TCP/IP or https.


One because it’s the law and two because the disabled can’t just make a choice and install Chrome.

However, while the site creator does have to meet the disabled halfway, the disabled person is responsible for having whatever type of equipment they need to make it work - ie screenreaders


If your website is full of divs generated by JS that are full of aria tags that make no sense, those tools don't have a chance. Most websites act this way as well. Even Facebook used to lock people out of their messages if you couldn't use a mouse, at least in the last time I checked (infinite feed + no way to skip feed via tab -> can't reach right panel).

Just do your job right. Not saying you should test some unique Firefox config but at least the default version is to be tested.

Hell, I've seen people here indicating that they just tell desktop Mac users to "install Chrome". Such carelessness is bad for business. Web development sure could raise its bar.


If you’re selling a SaaS app, you care more about the customer than you do about the user. The customer is the IT department.

For the longest, Amazon Connect’s - AWS hosted call center software - call flow builder only worked with Chrome.

Even for B2C users, using Chrome is not a deal breaker. If they are okay with using shitty Electron apps, they will be okay with using Chrome for Mac.


Because they are standards compliant and you aren't, and you are legally required to provide an unsubscribe service or whatever without undue barriers around it.


For unsubscribe - yes.

Everything else - no.

But if I am using standards and they have an ad blocker that blocks some of the functioning of my site, am I also required to test my site against that?


> Everything else - no.

I'd include _everything_ important in the "yes" category. If I cannot access the customer panel to update settings or notify them of a bug that is affecting me because I'm using Firefox ("works for 95% of users"), they're just not keeping up their end of the contract.

Remember, 95% excludes everything but chromium/webkit-engines.


If that 5% is 90% of cost to provide the service, forget it. Nobody is going to do a Herculean task to support a niche user.


Every SaaS company I’ve worked for has had a compatibility matrix where we say what we support. If we lost customers who were running a highly customized Firefox on Linux, so be it.

Every company decides which customers are worth going after.


Yes, sure, but 5% includes stock firefox, zero modifications, zero plugins.

Might still be a business decision, but it's like saying "we'll drop any emails that indicate a mail client other than apple mail/gmail/outlook".


While not that strict, see how far you get hosting your own email as far as not being rejected or automatically classified as spam


And I'd include that as well: if your server rejects emails because of your spam-decisions, you can't claim "we've never received that email". Either you don't use email for any legally-binding communication ever, or spam-filtering is a you-problem, not an everyone-else-problem.

It's not surprising that the strongest protections always happen on the unsubscribe links, but not on the subscribe-links. That just needs to be fined out of existence, just like "you can order with one click, but you need 50 clicks and a three-hour-conversation to cancel".


I don’t understand the “automatic” here-yes, reputation takes time to build, but if you run your own mail server with SPF/DKIM/DMARC set up correctly why is the default posture “block it” before there’s any reputation?

Just like other cases, I won’t accept that it’s “just lazy” on the part of big tech companies. They clearly know how to adjust their internal view/reputation of a domain once it starts being used for “misbehaviour” and spam such that they start blocking it.

Thus they could clearly start by not doing so-and, maybe, they’re “really touchy” about domains with no initial “internal score” such that if a new domain pops up and starts spamming people they catch it fast. Its not necessary to break open Internet protocols, though, unless they want the breakage.


It'll be interesting to see what happens if someone takes that argument to court.

One side of the argument is that Cloudflare places an undue burden. The other side of the argument is that without the CF protections, the service provider doesn't even have reason to believe the request is coming from a human being the law protects.


> and you are legally required

Where. It’s global internet we communicate via.


> If you look like a bot, how are they going to distinguish?

Some non-existant system of attesting that I'm person X (possibly through an e-ID card) who has issued a client certificate Y (cert chain, using my e-ID cert to sign) to be used with my device Z (presumably with a device fingerprint or IP range attached to the cert). Of course, this would mean no privacy, but that's not that different from being signed in through Google as an identity provider, we'd just shift the mechanism to be universal (like client certs already are). One of the options that would take more coordination than will probably happen (though very similar to some e-signature solutions in EU, which we already use) but I could see using something like that for a variety of professional/service sites, since signing in with the e-ID card directly is already a thing on some sites here (government sites, banking sites, utilities sites).


Okay. Do that globally. And solve the ddos problem as you’re on it. If you add transparent tls termination, edge, caching, dns… maybe I’ll have a look!

I had a guy like that working with me. Blocked every possible tracker, disabled javascript, used some niche browser, proton mail, and then complains that google doesn’t allow him to sign in. I get it, privacy and what not. But the guy was an outlier.

Some random blogs, product pages aren’t gov, most likely have no way to opt-in for gov eID (maybe they aren’t based in the EU), and they only care that their service is available fast globally and that they get ddos protection for free (plus some other convenience features).


> Do that globally.

We already do a simpler version of that with TLS and HTTPS, there are globally trusted root certs that ship with most OSes and browsers. It's just that we haven't extended the same approach to client certs and identity verification, instead having a bunch of walled gardens and governments running legacy methods of figuring out who someone is, as opposed to various eID mechanisms.

If I trust news.ycombinator.com because I trust ISRG Root X1, I might similarly trust John Doe's iPhone because I trust the government of France's CA, as a hypothetical, as long as the certification chain is valid there.

It's a problem that's technically solvable (say, in 20-50 years), but won't get done because good luck getting a bunch of governments to collaborate on that across the world. It's actually a surprise that we have TLS in the first place.


> If I trust news.ycombinator.com because I trust ISRG Root X1, I might similarly trust John Doe's iPhone because I trust the government of France's CA, as a hypothetical, as long as the certification chain is valid there.

There are a whole ton of privacy problems with this. I am happy to demonstrate anonymously that I am not a bot, but a random blogger does not need to know that I am John Doe, a citizen of France with national ID number 12345678.


We cannot get them to agree on cookie banners and you’re talking about something much more complicated.

Hey, by the way, would you trust some Chinese or Russian root certificate?

The question is irrelevant, frankly. Consider this: you’re living in Germany today. You trust the German government. They handle all your logins using that eID. What if in February AfD comes to power? Do you still trust the German government? Governments are formed by people. Different people have different interests.


> We cannot get them to agree on cookie banners and you’re talking about something much more complicated.

Another good example of something that’s technically feasible and not that complex, but was made infeasible due to either ignorance or malice, with all of the dark UI patterns and scummy behaviour.

> Hey, by the way, would you trust some Chinese or Russian root certificate?

Most people already do: https://chromium.googlesource.com/chromium/src/+/main/net/da...

For example:

  CN=CFCA EV ROOT,O=China Financial Certification Authority,C=CN
  CN=GDCA TrustAUTH R5 ROOT,O=GUANG DONG CERTIFICATE AUTHORITY CO.,LTD.,C=CN
  CN=UCA Global G2 Root,O=UniTrust,C=CN
  CN=UCA Extended Validation Root,O=UniTrust,C=CN
  CN=vTrus ECC Root CA,O=iTrusChina Co.,Ltd.,C=CN
  CN=vTrus Root CA,O=iTrusChina Co.,Ltd.,C=CN
If there’d be an issue of not wanting to support a certain country, then removing such a group of CAs from a store would be trivial for a particular service, same as with the above.

Plus, the opposite is also viable, if for example the Russian govt. wanted to allow anyone to verify whether particular requests come from their citizens, they might also run their own CA akin to https://www.bleepingcomputer.com/news/security/russia-create... except that the attack vector would change from MitM to fake identities being issued by them as needed (but since the server is the one doing the verification, it might as well drop the CA when desired).

> What if in February AfD comes to power?

Revoking the eID and anything dependent on it would be akin to your passport being taken away.

Essentially the modern day digital equivalent of getting your Google account banned by some bot, if you use that account for auth in a bunch of places.

Fundamentally, that’s no different from the reality that we already face - my regular eID could also be taken away if my own government felt like it, same as with my bank account and other assets.

Client certs themselves are nothing new, same for PKI. It’s a cool technology that could but presently cannot solve the problem of client identity globally, because we just can’t have nice things and order.


> Revoking the eID and anything dependent on it would be akin to your passport being taken away.

Is it? If my eID is used for logging in to my bank and said eID is revoked, I can no longer log in to my bank account. That’s completely different than a locked up passport.

> Essentially the modern day digital equivalent of getting your Google account banned by some bot, if you use that account for auth in a bunch of places.

Use a custom domain, don’t make your kingdom dependent on the gmail.com address.

I don’t know, for me the perfect amount of government oversight is “as little as possible”. There’s zero need for the government to mediate between me and my bank, or some random service provider on the internet.

What you’re describing sounds like a fun technical challenge assuming a perfect world. For example: who decides which countries’ certificates should be revoked? Who decides who is the rogue one? Even that is stretching it too far. Can I simply download a browser without some selected certificates? If the technology is so great, why isn’t it widely adopted today

Those are all rhetorical questions. You don’t have explain PKI to me.


> Is it?

Pretty much the same failure mode, just with different immediacy. No more travel, no more ability to start using new banking services, no more proving identity for becoming employed, pretty much anything that needs you to provide valid governmental ID (ID card or passport) and doesn't accept alternatives.

On the opposite end of that, both those services might accept something like a driver's license and the banking service might allow you to log in with their app, or a similar identity provider as a backup.

> There’s zero need for the government to mediate between me and my bank, or some random service provider on the internet.

Who else should we depend upon for verifying the identity of someone? Because currently it's a hodgepodge, especially when some places treat the equivalent of an SSN as a secret or have other half baked mechanisms, whereas in actuality it's a problem that's been solved far better, the same way how e-signatures work here when a single competent authority implements them well (certs on the e-ID card, you choose what to sign, but there's both data integrity and non-repudiation, a service that everyone integrates with and it is basically treated as a commonplace utility).

> What you’re describing sounds like a fun technical challenge assuming a perfect world. ...

Yeah, that's about it. Have a good one!


Between what you described and having to run a vaguely standard browser config, I'll take the latter, thanks.


Ok, what does the venn diagram of:

1) People who anonymize their IP, use Linux, a browser with noscript, etc

2) People who are OK with having a government issued digital id and having to use it to access the internet

...look like, in your opinion?


Well, proof of having an ID can be done anonymously. Cloudflare even worked on a system for that kind of thing.


A non-citizen living in Germany without the German eID because they’re not a citizen. Their country of origin doesn’t have any of that. I guess they don’t exist in that setup? Seems like a steep hill to climb on to solve some random login with captcha problem.

Binding login interaction to some government issued id…who’s entitled here.

Sounds like throwing a baby out with the bathwater.


Then have them go through the captcha process that already exists


Yeah, this is at least being discussed now for eID. Getting it to a point where it is actually usable for everyone and trusted by everyone will not be easy though. But even in the best case, this would cover maybe 5-10% of internet users in 5 years. What do you do with the other 90% ?


I don't buy this because using Chrome is what most bots probably do right? Headless chrome is easy.


I honestly don't see what's so hard about a bot simulating "the norm" within the margin of error. This cat-and-mouse game is just like a GAN, the end result is indistinguishable even by a bot.


It depends on the defences. It starts trivial - just make a http request. Then there's http version, user agent header, other headers, header ordering, cookies, TLS ciphers, session resolution, timing, behaviour for page resources, ... and so many other things. It takes time, even if you order headless chrome.


Bot authors are lazy and won't until they have to.. once you do, you can then pretend they aren't bots and include them in the engagement numbers you feed prospective shareholders.


Agreed. From my past experiences though, a very good chunk of them will give up once there is a resistance. Basically, you want your bot protection to just be a little better than your competitor. Then the bot author will target them instead, because of the path of least resistance.


Outrun the friend not the bear? Hehe


it is discrimination


Only if enough are discriminated against that it affects the bottom line.


The problem is that any solution so far proposed for this is very privacy-unfriendly.

For example, Google proposed https://github.com/explainers-by-googlers/Web-Environment-In... and this was shot down by privacy advocates (for very good reasons).

So basically the choice for website operators is either to fight the bots and accept that their service will be unusable for some subset of their users or not fight the bots, which will lead to their service becoming unusable for everyone.

More and more, you see services pushing you very hard towards using their app and the reason is that with the app, they are able to actually verify that you are likely not a bot (or rather, in reality, that at least the app is running on an actual physical device, mobile phone bot farms are unfortunately also a thing).

As for Cloudflare - they offer it as a service, so when the website operator has a choice between using them or allocating several engineers for bot-fighting, why would they not just go with Cloudflare? Doing it yourself can be slightly higher fidelity, as you know your customers better, but it is also a lot of effort which could be better spent elsewhere.


> either to fight the bots and accept that their service will be unusable for some subset of their users or not fight the bots, which will lead to their service becoming unusable for everyone.

2/3 of the issues OP listed would not make the service unusable for anyone if the botcheck were removed. 1. What would be the problem with allowing "bots" to opt out of receiving marketing emails? Why do I need to be a human to tell you to stop spamming me? Who is running such a bot, for what purpose? 2. What would be the problem with allowing a "bot" to log in to an already-verified human account a single time?

The only situations where you actually need to confirm that a user "looks human" is for repeated connection attempts in quick enough succession to matter (DDoS prevention), or when they want to do something that someone would actually write a nefarious bot to do (mainly just creating posts/messages visible to other users).


Just an idea, what if malicious bots started unsubscring thousands of email addresses to harm your business.

Even if you send a confirmation email afterwords that's potentially millions of emails you are sending because of bots.


> what if malicious bots started unsubscring thousands of email addresses to harm your business.

GP said:

>> need to confirm that a user "looks human" is for repeated connection attempts in quick enough succession to matter (DDoS prevention)

And even in that case, you could implement other solutions. For example, for unsubscription links, you could pass a "token" in the query string that "verifies" that it's the address' owner unsubscribing. You could generate such token either stateless (JWT, for example, then verify it) or store it somewhere along with the address.


Why "fight the bots" anyway? If software that is acting out the will of some humans somewhere is retrieving static contents, what's the big deal?


I deal with this fairly commonly, presumably because I use linux, and we all know only botnets use linux. Occasionally with cloudflare I'll just get summary rejection and supposed blocking of my IP, but either it's summary rejection or a pass without challenge.

Recently I had to deal with this for alibaba just to look at something, which I usually just use torbrowser with, and finally gave up as I couldn't pass the challenge. I suppose I shouldn't be surprised at that though, they trust me as much as I trust them.

The worst is usually adobe and cookielaw with all their related tracking crap, where I can't even get the captcha to render as it's so many layers buried in scripting I can't enable enough sites between ublock, noscript, privacy badger, and firefox strict modes. I treat adobe like malware, but unfortunately things like albertsons.com for groceries and other mega companies love to use it, and their sites literally do not work without allowing their heavy scripting/tracking.

There are other usually smaller captcha players that I haven't been human enough to pass with, I forget the names of the stupid to shame, but a few when I see them I recognize to just close the window and forget about whatever it was I was looking for there (like twitter/x).

Hooray commerce!


I visited albertsons.com out of curiosity, but I was instantly banned. Even using an unmodified Chromium browser, I couldn't access the site. It's ridiculous what's happening on the internet today.

The error: ``` Access denied Error 16 www.albertsons.com 2025-01-03 09:30:00 UTC What happened? This request was blocked by our security service Your IP: xxx Proxy IP: xxx (ID xxx) Incident ID: XXX Powered by Imperva ```


Their Imperva WAF usually challenges me repeatedly during use trying to buy groceries from my pc, and most of the time I get tired of having to disable every security extension I use with Firefox to use Albertsons because of their shitty website. Never outright block though.

Might be worth checking some enterprise threat lists for whatever IP's your popping up on (ie Imperva and Cloudflare), or something uniquely fingerprints you from your browser. I use multiple extensions to block whatever they each can, and even I'm not treated that badly as you for wherever you are coming online from.

Here's Fortinet's you can check your IP against, they all tend to roughly use the same lists eventually: https://www.fortiguard.com/iprep


Immediate bans might be related to the country you're in. This is a US retailer, and there is zero reason for someone outside of the US to visit that site. Blocking foreign visitors allows them to ignore GDPR, for example.


In this case, they may display a message like: 'This page is intended for USA visitors only. Our services do not operate outside the USA.', but no, they say you are banned because just...


My main desktop for the past year has been Steamdeck with linux. And don't get any excess Cloudflare challenges.


Nice idea! How's that working out for you? Stock OS? Bazzite?


Stock. Browser (Chrome/Firefox) doesn't have hw acceleration for video decode. But other than that it's fine. Fast and silent. VS Code and Jetbrains tools work fine.


>...when I see them I recognize to just close the window and forget about whatever it was I was looking for there

This is the way.


Yes. I wrote about this on my blog six months ago [1].

CloudFlare has positioned itself as the doorman of the Internet, deciding who gets to visit shitty websites written by AIs and who doesn't. Every time I try to visit a website and get blocked by this company and its unnecessary services, I congratulate myself for avoiding yet another terrible website and move on with my life.

[1] https://ido50.net/content/what-chafes-my-groin-9.html


The doorman for the internet. well said. Someone need to study how this is likely the most successful marketing campaign ever for a cloud provider.


I don't think they needed much marketing? A lot of website operators want bot/DDoS protection, and cloudfare offers service which works (at least for overwhelming majority of users), and is absolutely free.

Offering free stuff which works and that many people want is how internet companies get big.


Indeed, and cloudflare has also improved search engine effectiveness. If I'm looking for the answer to a technical question and four out of the top five hits are cloudflare captchas, the primary source is readily identifiable.


It seems a bit shortsighted to think that CloudFlare only does this for 'shitty websites written by AIs'


I thought it was obvious I was being facetious.


It wasn't to me, apologies.


It's ironic but I was having terrible problems accessing archive.today when I was using Cloudflare DNS (1.1.1.1) that cleared up when I switched to either my ISP's provider or Google's 8.8.8.8. I was not the only one

https://news.ycombinator.com/item?id=38063548

What's funny about it is that as a human I get tormented by those things all the time but I have been writing bots since 1999 and have yet to have had CAPTCHAs affect a webcrawling project in a big way: for instance I have a bot that collected 800,000 images from 4 web sites since last April, at times I thought they had anti-bot countermeasures but I realized that when they were having problems it was because the wheels were coming off their web site (don't blame me, that is 0.03 requests/second and are not parallelized and pipelined like the requests from a web browser.) I'm also prototyping one that can look at an article like

https://phys.org/news/2025-01-diversifying-dna-origami-gener...

see if there are links to journal articles in there, determine if the articles are Open Access and pick out an image for social... so far no problems. But if I want to pay my electric bill there's a CAPTCHA -- I mean, what kind of bot wants to pay my electric bill? (Kinda seems like it is asking for a lawsuit in this day and age if it prevents anyone 'differently abled' from accessing essential services...)


That's not because of spam blocking (directly), it's because of a particularity between how the operator of archive.today wants to handle DNS and how the Cloudflare resolver handles it.


> I mean, what kind of bot wants to pay my electric bill?

None, but they do want to use your electricity company's credit card payment facility to test stolen card numbers.


> I was having terrible problems accessing archive.today when I was using Cloudflare DNS (1.1.1.1)

That's because that web site returns bad results to Cloudflare DNS, ostensibly because they take issue with the way it handles EDNS0. The fact that it fails to work is a deliberate choice by the site operator; it isn't Cloudflare's fault.


That's oversimplifying a bit and missing some critical information.

Cloudflare wants to "protect" people from exposing even their general region. This has the side effect of making CDNs that aren't Cloudflare work worse. Cloudflare are being dicks because they do to others what they wouldn't want to be done to themselves, or what they themselves don't do to themselves.

It's not even that people are choosing to opt in to Cloudflare's bullshit. If you use Firefox in the US (and many other areas, but the US for sure) and you haven't manually configured Firefox or set up a canary domain, all your DNS lookups are going to Cloudflare, and they're using that to make other CDNs work less well. That's definitely shady and definitely bad on Cloudflare's end.

I'm glad some people are taking a stand.


the russian archive site and cloudflare have been having a dispute for a while now


Russian?


Please do not use that term! I cannot fly! I don't believe in that sort of thing, either. I'm libertarian, and would rather not sue over much of anything! Especially something that would lack standing. Oddly enough, I haven't been interrupted by CloudFlare too much. I do use Firefox on Windows, but haven't gotten into Linux as of yet. Although it might be fun, I'd probably break it too much lol! I do run adblock, mostly for accessibility reasons. I don't want ads all over the page, when I'm trying to navigate. That makes the web suck a whole lot less! I do like RSS, I wish it was still supported in the browser, without an extension.


(1) I was working for a small town web design shop in a town dominated by two higher ed institutions circa 2005 where I was finishing up a system which handled applications for an internship program run by the NSF. (e.g. complex forms, scanners to stop people from uploading CVs in Word that are full of malware, etc.) I was talking w/ the principal about how bad the usability was of applications in higher ed and that I was surprised that there hadn't been a lawsuit to the effect that "I tried to apply to organization X and couldn't because their website was screwed up" and he said that he thought they'd probably been settled out of court.

I work at the Uni now and circa 2015 we had a lawsuit against us because we made people use terrible quality applications that weren't accessible. I'd make the case that that sort of organization which has a rigid social hierarchy (e.g. grad student, postdoc, assistant professor, associate professor, full professor, department head, provost, ...) finds it close to impossible to confront quality problems that it finds invisible. (e.g. if you submitted a bad paper to a journal or had sex with an undergraduate it could understand that but a web site could set your computer on fire and they wouldn't see a problem with that.)

Since then all higher ed organizations feel a lot of need to offer accessible applications. My unit sells a subscription service to a data product and in sales talks and other conversations with our customers we find accessibility is a priority so it is a priority for me as a web dev.

(2) Don't get me started about RSS. I think it is great, kinda. Fir $10 a month I can pay Superfeedr to scrape 110 news sites and send them to my web hook which queues them in SQS and lets my RSS reader YOShInOn ingest them at its own convenience. I'd like to subscribe to 2000 or so independent blogs but don't want to pay a $100+ month scraping bill.

Could I write my own crawler? Sure! But polling is for the birds. You really want to get a ping just when the event happens (ActivityPub? PubSubHubbub? AT Protocol? XMPP?) but instead you have to poll. There are two kinds of polling: (a) too fast, (b) too slow. Should I run it at home over my slow ADSL connection (is my wife having trouble using the internet because my crawler is having a bad day?) or should I run it the cloud where trying to save $5 a month on my bill could cause EBS volumes to go swap crazy costing me $500 a month? It's awful for people who run feeds, see

https://rachelbythebay.com/w/2024/05/27/feed/

although she should (a) just get a CDN and get over it or (b) give up on RSS. Sorry, people write stupid stateless crawlers with curl and making your crawler stateful enough to respect her silly 429 protocol makes RSS no longer a simple protocol.

On top of that people keep failing with the same failing user interfaces for RSS readers that have been failing with 1999 with no insight that "people tried that in 2001 and it failed". People like Dave Weiner have no insight why the world couldn't care less about RSS because they just won't recotnize there are problems.)

(e.g. if you gotta know, YOShInOn works like TikTok... I never "mark as read", it doesn't show me little windows that show me the top N from 20 different sites, none of that.)

(3) If it's your electric bill it really is an essential service that there is no competition for. Frequently markets work, but not in that case, even if Enron was able to fool some legislators that they would work in that case for a while.


Polling for time sensitive things sucks. But for RSS, why not just poll once a day? Even if you're polling once an hour, a 304 response is, what, a few hundred bytes?

I just checked my own server logs, and HTTP 200 responses were 12.7% of the total requests against my rss.xml. Which is suboptimal I guess (I haven't made a single post this year) but isn't outrageously terrible.


Maybe it's lame but I sure wish my agent could know the exact second that a new post gets posted to

https://www.righto.com/

and automatically posts it to HN. Sometimes having an average score/post doesn't seem enough to me and I think of adding a population of high scoring posts. Like that blog. Or those articles that have been posted 20 times before and gotten between 200-1200 points each time and will probably do so if you post them again.


Sure I believe in accessibility, just not lawsuits, yes two different things! If I don't like the way a service works, I cut ties with that particular service, and don't complain. Sometimes I'll contact them first, to advise them they have an accessibility issue, but I'm nice about it. I did this last year actually, I was testing feeder, I contacted them, to let them know I would be testing the service. When it really wasn't working for me, I needed them to make some tweaks, when the tweaks they made didn't work for me, mostly change the tabbed navigation that I wasn't using, I left. That being said, I like RSS a lot! However RSS readers these days leave a whole lot to be desired! I've been looking for one since 2013. I did use Miniflux for a while, until the service that was managing it for me shut down. I didn't learn terminal when I was young lol. I also have chronic migraines, so not sure if I could try and fix that now.


Cloudflare is so embedded into so many important services (like some other companies, including Google), that they need to be thinking of their role as having some government-like responsibilities.

For example, for starters, Cloudflare and Google need to find ways so that individual people who're wrongly being locked out of services by the company, have some way to get that unlocked. Not "sux2bu we dont do support bro".

(Then they can start thinking about the next step, which is due process, and what it means to wrongly lock out someone in the first place.)

That said, as an immediate pragmatic matter, one debugging tip with your Firefox is to go to the `about:profiles` URL, and temporarily create a new profile, and without using any Firefox sync feature, and see if Cloudflare lets you through, and then incrementally add back in your extensions and preference customizations, and see if/when CF stops letting you in. (Not that it will necessarily identify the sole and exact trigger, since they might be using scores of multiple factors, but it will be evidence of one thing that pushes it over the edge. And maybe get you to a compromise setup that lets you do your work for now.) Also helpful is to have alternate browsers installed; personally, I keep Chromium installed, as my "violate me every possible way, if you'll just let me access this one page/site I really need right now".


I had similar issues as an (also heavily customized) Firefox user, but was able to fix it by installing Cloudflare's Privacy Pass browser extension.

It seems ironic that as a human I can't seem to reliably prove I am a human with a realistic amount of effort via these systems, but having installed a specific automated browser extension does?

I am not a fan of Cloudflare and don't like the idea of running their software on my computer, but it seemed like the only options to continue using the internet at all.


I didn't know that extension existed, so after failing to fix it by reinstalling Firefox and removing extensions, I just gave up and installed Chrome.


I can't use any of the kerbalspaceprogram.com domains because of improper discrimination against IPv6 clients triggered by CloudFlare.

    Error 1015 Ray ID: .... • xxxx-xx-xx xx:xx:xx UTC
    You are being rate limited
    What happened?
    The owner of this website (wiki.kerbalspaceprogram.com) has banned you temporarily from accessing this website.
This sort of monoculture creates an Orwellian SPoF.


I don't think it's an IPv6 problem. IPv6 clients are more static than IPv4, which is usually shared amongst many clients (at home) or at the network level (CGNAT).

It could be the address is being reused - is it home, cloud or corporate? Have you tried different browsers? Incognito mode?

I have an IPv6 block at home and have no problem accessing that site.


That isn't "triggered by Cloudflare". The operator of the web site has deliberately configured it to block your IP range, and Cloudflare is obeying those instructions.


Cloudflare owns kerbalspaceprogram?


No, wiki.kerbalspaceprogram.com is a customer of Cloudflare, but the outcome is the same.


Then ask them to disable Cloudflare.


good luck with that


Exact same situation here. Linux, fairly funky firefox setup, eventually couldn't use half of the internet without hitting CF prompts, often wasn't able to get around them.

I wound up removing / reinstalling firefox...same exact setup otherwise. No more cloudflare (or vastly fewer) prompts. The internet is usable again.

Hope that helps.


I had to switch to Chrome. Reinstall was not enough for me.


People are focusing on your very non-standard setup, but I've experienced this - less than you to be sure - on a standard MacOS setup with Firefox and only uBlock Origin installed. If I switched to Chrome without uBlock Origin it worked. This was on the English National Ballet's ticketing website.


Same problems here. Mac OS with firefox + ublock origin (and a dns based ad blocker) jams me up. Switching to Safari (with dns ad blocker still on) makes it work

Has become increasingly more common in the past few months across several sites.


> - The "unsubscribe" button in Indeed's job notification emails leads me to an impassable Cloudflare challenge.

Maybe indeed could be held liable here? From the can spam act (if you're from the US):

> You can’t charge a fee, require the recipient to give you any personally identifying information beyond an email address, or make the recipient take any step other than sending a reply email or visiting a single page on an Internet website as a condition for honoring an opt-out request.

https://www.ftc.gov/business-guidance/resources/can-spam-act...


this nevertheless happens all the time. i have an old linkedin account i haven't logged into in years and can't be bothered to dig up the credentials so one of my e-mails gets stupid "network updates". one must log in to disable these and navigate to some obscure settings page in one of the most heinously overcrowded UIs on the web.

so i just flagged it all as spam and hoped it hurts their deliverability a little.


Honestly I click an unsubscribe link but if it requires me to complete a survey or fill out a form, I just nix the tab and spam filter the email. I'm nobody's fucking admin assistant and my time is valuable: you know my fucking email and could easily add it to the think, or at the most, ask me to type it into a box if you MUST. Anything more than that, if I have to manually opt out of "types" of messages or whatever, nah. Fuck you.

I didn't ask for your fucking emails and I sure as shit am not going to do the homework you're assigning me to make them stop.


If the survey has text fields and I have enough spite left in me I fill them with "[object Object]" in the hopes it makes someones day more miserable than mine.


takes notes


Yep, I just spam filter the E-mails now. If that act adds 0.0001% to that sender having future E-mail deliverability problems, then all the better. If it's commercial or political and I didn't explicitly ask for the sender to E-mail, then it's spam.


It does! Reporting as spam will cause them to have issues if enough people do it.


I do NOT like it at all but I just want to show a way how it works with Cloudflare and to make it painless with them. Basically fully assimilating to them because Resistance is Futile ;)

1) Privacy Pass Extension

Install Privacy Pass Client Extension in your browser, here for Chrome https://chromewebstore.google.com/detail/silk-privacy-pass-c...

2) Use Cloudflare Warp (which is a VPN by Cloudflare basically, it's free):

https://one.one.one.one/


The privacy pass extension still requires you to pass a cloudlare turnstile which is impossible in some browser configurations. E.g., if you disable browser performance-debugging/timing features (these used to be a vector for Spectre timing attacks).


I'm really afraid of what kind of internet we'll have when these kinds of un-diagnosable un-appealable false-positives are not just transient blips, but become metadata companies use to blindly and permanently kill off accounts on other services.

I think it may have been what happened my since-2010 Reddit account was mysteriously killed a couple years ago, and literally the only cause I can think of is that I might've used the wrong public wifi for an evening.


I absolutely hate cloudflare for the same reason you have. Besides traveling and using a VPN, I like in Hong Kong, a country that many sites have decided to block completely. It's very frustrating that cloudflare easily enables those kind of blanket bans for no reasons.

Cloudflare is the enemy of open web.


Cloudflare works much, much better than Google - Google captchas for me, on Tor, are flatly impossible, always. They never let get through, no matter whether you get them right or wrong. You always get "try again".

The problem I do have with CF is their captchas seem to require human interaction on the page, and this makes getting through them problematic when you open half a dozen tabs, and each loads a CF captcha, and you have to move the mouse around for ten seconds just to get the captcha to load, and loading is not reliable. Often you need to reload the page. It's this type of performance, and poor performance, which is breaking web-pages for me.


But at least with Google captchas you can use AI to solve them. I use the buster captcha extension to solve them. It moves the mouse around like a human and solves automatically. I pay for captcha solvers for hcaptcha which is worse but cloudflare is just cancer. It’s made the web unusable


Ehhh... maybe...

Last week I had a run of (legacy) Cloudflare captchas on sites protected by CF to solve of "select all the boxes with motorcycles in", and despite doing it fastidiously and correctly (although I never know how to handle the boxes with like 3 pixels of object in but are otherwise clear), I had to do it like 5 times with different images, until suddenly it was happy.


legacy Cloudflare captchas?

I thought they eliminated them back in 2023? Their announcement is pretty clear on them:

"Cloudflare will never issue another visual puzzle to anyone, for any reason."

https://blog.cloudflare.com/turnstile-ga/

Are you sure it's not fake? For example archive.is sometimes sends me orange-colored CAPTCHAs (with "select all the boxes" style) that are never accepted; but if one looks closer at them, it actually never says "cloudflare" on them anywhere, nor there is a logo (it does this because it has a long-standing feud with cloudflare re users' privacy).


I thought they had as well, which was why I was surprised to see them.

I can't remember the site I saw them on, so I don't know for certain, but the site was definitely protected by Cloudflare, and I'm not really sure what you mean by "fake" - they were definitely CAPTCHAs with image tiles, but I guess I don't know for certain they were coming from Cloudflare servers.


How did you know they were Cloudflare and not some other service? Was there a "cloudflare" word or their logo?

HCaptcha is still a thing, it has "find all dogs": https://developers.nopecha.com/recognition/hcaptcha/

I get flagged by Google's reCaptcha all the time, it's super annoying: https://developers.nopecha.com/recognition/recaptcha/

Archive.is made their own check page (probably based on HCaptcha), which uses same color scheme as cloudflare and same general design.. except if you look closely, you'll notice there is no word "cloudflare" anywhere on page, it's a homegrown solution: https://www.reddit.com/r/SaintMeghanMarkle/comments/18c3ea7/...

For some weird reason, people like to attribute every captcha to cloudflare, even if they no longer offer this service...


That sounds like a feature. Tor is for abuse, so you don't want Tor people hanging around on your page.


No it isn't. I discuss politically sensitive topics through it basically every other day, because by doing it directly in my country you will quickly end up in prison. (No, there's no scarecrow of the day involved, just discussing things you take for granted in your liberal democracy.)


> Tor is for abuse

No. Tor is for anonymization. Some might use that for abuse, but that is not it's raison d'être.


Unfortunately, your setup makes you look like a scraper: no history for Cloudflare to identify, the sort of browser / OS config someone would use to homebrew an automated "I sure am not a bot, look at how authentic my user-agent is!" bot, and so on. If you also have JavaScript disabled and clear your cookies frequently, Cloudflare can't fingerprint your machine to know you passed a trust-check in the past.

Maybe keeping a heavily-sandboxed Chrome in a VM for situations where Cloudflare is getting in your way might help?

(In the large: this has been an issue a long time coming. Quite a bit of cyberpunk predicts the future where the web bifurcates into the "regular" web that is sanitized, corporate, controlled, and used by most people... And the "everyone else" web that is not, with all the pros and cons that entails. The tech has evolved to the point that companies that want a service provider "keeping the bad guys away" for them can pay to have that done, at the cost of false-positives... But at their scale, the false-positives may not matter to them).


My workaround for this as a person who travels a lot was to buy 2 raspberry Pi’s and put them at my family houses in different countries and use Tailscale on them as exit nodes, behaving like my own VPN. The residencial IP address makes things a lot easier when connecting from random places.


I have experience bypassing these.

The primary cause of this is most likely any kind of 'optimizations' you have in your browser (or missing fingerprints).

If you want to 'bypass' these I recommend removing any use of Proxy[1] (via extensions). You should also look into disabling any kind of forced backgrounding. Make sure service workers are working.

1: They catch Proxy usage by using exceptions and analyzing the stacktrace. I assume you know what a javascript proxy is, but incase you don't: It's something that allows you to override any kind of object function such as navigator.hardwareConcurrecy.


> They catch Proxy usage by using exceptions and analyzing the stacktrace

That is really clever, I am guessing this is why various browser automation companies are using custom forks of Chromium.


I appreciate you bringing up this issue about the Cloudflare challenges making it hard to browse. I had a similar experience where I couldn't access jsfiddle even without using a VPN. As a result, I switched to a different platform for my coding experiments.

JsFiddle used to be my favorite for quickly testing out code snippets. It's a shame that due to Cloudflare hurdles, I've stopped using it and don't plan on going back.

It may not be much but as more websites and businesses lose genuine web traffic like this, Cloudflare might eventually listen and fix this mess.


One concrete thing we can do is to stop seeing Cloudflare as an easy, unproblematic solution. Bring up issues like this when people suggest using it.


I'm experiencing the same issue which is definitely exacerbated by straying from a 'default' configuration e.g. using a custom browser screen reader, browsing from Brazil, using a VPN, using Firefox. I think eventually I'll be completely locked out of the 'mainstream' web


Yes, I run into it from time to time. I just move on. If someone is going to make their website inaccessible to me, I'm not going to bend over backwards to try to work around that.

Incidentally, since I configured DNS over HTTPS in Firefox, using Cloudflare's DNS, it seems I see this much less often.


AWS WAF is even worse. I recently moved from Australia to India, and quite a few high-profile websites are now completely inaccessible to me because WAF seems to be legitimately broken. Two such sites: https://officeworks.com.au/ and https://centrecom.com.au/. You successfully complete their annoying thingummy, and it redirects you… to the same Human Verification CAPTCHA. This has been the case for at least half a year, so it’s not a recent breakage.

If I tunnel via my VPS which is still in Australia, then I can access it.

But complete blocks via Cloudflare have also been a problem: I had to do something with VicRoads as part of selling my car, and was blocked outright when I got to the actual form page. Had I not had my VPS in Australia, I don’t know what I would have done.

My IP address is massively shared (CGNAT) with plenty of botnet around, so I’m frequently troubled by Cloudflare, but not often outright blocked, and if challenged rather than blocked, I’ve never had any problem with it. Linux, Firefox.


As another aussie expat abroad, leaving a box behind at my parents place for an Australian residential IP has got to be one of the most unexpectedly great things I've done.

Wireguard/Tailscale and my parents having access to cheap renewable power are the real enablers ofc.

To anyone moving abroad in the near future - leave a box behind with your parents/close friends, it's well worth the trouble if they're ok with you occasionally mooching some bandwidth. You absolutely won't regret it


And don't be tempted to run any upgrades on it until you come back :)


Yeah, I’ll admit there’s some paranoia about losing access after a botched upgrade.

I’m considering investing in a https://tinypilotkvm.com/, but that can wait till I’ve lost ssh at least once. I’m not hosting aws on the thing so I can afford to play it fast and loose :)


imo it's more flexible to just have two boxes, especially if they're cheap.


Oof. Been there.


I've had to give up obfuscating my user agent because Cloudflare becomes nearly impassable as a result, and Cloudflare seems to own most web traffic now.


If you can't pass the captcha you have to ask yourself, are you really a human being or have you just been programmed to believe that you are?


>I use a heavily customized Firefox config on Linux.

This is probably the cause, especially if you're doing stuff like spoofing user agent. It's not cloudflare "cracking down on privacy" or whatever either. Unmodified tor browser passes turnstile challenges just fine.


It's up to users to choose their user agent.


And it's up to site owners and website security vendors to choose which user agents to admit.


No, that's discrimination.


i recommend everyone test the web with TOR to see how dead the public internet is. Reddit won't respond. Many sites have a 10-minute captcha challenge (e.g. substack).

So many sites have deployed countermeasures like Cloudflare, but they aren't actively monitoring the failure mode on those countermeasures.

The web is on it's knees and these countermeasures are another nail in the coffin if we don't act fast.


Why would anyone care about how a site displays on TOR aside from privacy nerds?

The average internet user doesn't even know what TOR is. Though they might have heard the words "Dark Web" once or twice.


it's a canary


> Cloudflare challenges have made large portions of the web unusable for me.

I guess the best web experience is when one filters Cloudfare, Google and Microsoft at the firewall.


Amen. Another fun one is logging into bank and government sites while roaming... with sms delivered intermittently and with a 5 minute delay.


If it is triggered by the customizations you did in Firefox, then running a fresh Firefox in a container might help:

    docker run -it --rm -e DISPLAY --net=host -v $XAUTHORITY:/root/.Xauthority -v /tmp/.X11-unix:/tmp/.X11-unix debian:12-slim
Then inside the container, run:

    apt update
    apt install firefox-esr
    firefox


The suggestion you should have to bend over backwards for shitty software like cloudflare is bad enough; but if you were going to surely creating a new browser profile is far easily than spinning up a debain docker image, updating it and the installing Firefox and the running it?


what is the advantage here over just running 'firefox -ProfileManager' and making a clean profile?


All host info not accessible via X11 protocol is hidden, for example font list, is replaced with generic one.

For even more protection, run VNC server with common resolution in the container and connect to it using VNC viewer. In this case firefox provides a super generic profile (latest debian with mesa GPU), making this browser very hard to distinguish from others. This has some downsides however: First, you cannot resize window. Second, a lot of actual bots use same config, so it might be blocked.


mullvad browser is pretty much this, but without messing around with containers. One fingerprint for all users, with the same font list, resolution, canvas behavior, etc.

https://mullvad.net/browser


looking at https://mullvad.net/en/browser/hard-facts , Mullvad browser is much more extreme: many APIs blocked, always incognito mode... I would not be surprised if this blocks some sites.

the container approach on the other hand is bog-standard firefox.


Isn't it suspicious bot-like behavior to only have the bare minimum fonts installed? :-)


To be fair, Firefox out of the box prevents against font fingerprinting more than Chrome, it's considerably easier to get Firefox to run in a docker container and pass all the client side challenges than Chrome in my experience, you still have a valid point though.


OP mentioned that they run a heavily modified browser, I think it means compiled with changes - docker means stock Firefox


Cloudflare puts challenges on their abuse contact page and rate limits it to much slower than human speed. It's also still broken after years in that you can't report abusers who register domains through Cloudflare and/or host their DNS using Cloudflare.

They really don't want feedback from people who don't pay them.


It used to be just for profit companies web dev's ignorantly putting themselves behind default cloudflare deploys and blocking everyone. But now big academic players like science.org/aaas elsevier and other publishers and individual journals are and I can't even read scientific papers anymore. Even sillier is the RSS/Atom feeds science.org ran have the same cloudflare rules so all actual feed readers were blocked (support told me only real feed readers as a service like Feedly corporation were allowed). It took me months of email back and forth to get them to realize their error and get to someone who could fix it. And that is what I consider a good response. Most just ignore the email.


Everyone reading this should start to contact websites/companies who use cloudflare and tell them in simple and few words that it's a problem and link them to a video or article that explains more, maybe even to this HN topic. We are not many, maybe 1-2% of their users/customers I keep reading people saying but I have in the past been able to get big tech companies to change to a friendlier tech. You would be surprised how effective it is to contact them about it. Maybe they have a tech support who already has same opinion as you but they can't make any change until a customer makes a complaint about it, then they happily see it as their opporunity to finally make a change.


I get this all the time with firefox on linux + ublock origin extension. Often ending up with that blocked ip page.

I mostly shrug off and just avoid visiting that kind of sites again. For an unsubscribe challenge I just copy paste the url and visit it using firefox focus on my smartphone on my mobile connection.


I wish we could popularize some extension that pays a penny per page load or something using some shitcoin both as a means to support our favorite sites but also to validate that I'm not a bot, or at least if I am, I am willing to spend a lot of money in a DDOS that goes directly in your pocket


Only the expensive bots with residential IP's and mechanical turks can survive, humans be damned.


I'd expect this to increase with the proliferation of AI Crawlers and scraping becoming easier with AI.


CrimeFlare is not interested in these problems for the users. If you have access to the hosting side, you can adjust the bot score for specific connections/clients. But consumers don't matter to CF so apart from jumping through their hoops, there's nothing better you can do.

Unless you accept the racket of course, start paying them and proxy your traffic through the CF workers https://github.com/pellaeon/cloudflare-worker-proxy and magically most barriers will disappear.


>Unless you accept the racket of course, start paying them and proxy your traffic through the CF workers https://github.com/pellaeon/cloudflare-worker-proxy and magically most barriers will disappear.

Source this actually works? ie. that using cloudflare workers allows you to bypass cloudflare protection?


https://jychp.medium.com/how-to-bypass-cloudflare-bot-protec... and many other posts. Haven't looked into this in a while, so can't tell you exactly how effective it is today. (Definitely corrects the high bot score of your IP though)


Sounds like all it does is make your IP reputation slightly better than tor, which is a pretty low bar to cross. You'd likely get the same effect from using any other VPN service, so it's not exactly evidence that cloudflare is running a "racket" with its worker product. The linked blog post even touts the fact it's free as an advantage. Rackets typically aren't free.


You also change the headers / TLS signature, because it's their worker doing the connection. That covers quite a lot already.

The racket is not in the workers themselves, but rather cloudflare both protecting from internet abuse and protecting sites which sell the abuse services. (For example hosting WebStresser) I meant that by giving them more traffic and accepting that as a workaround, we'd be saying "I'm ok with that".


>You also change the headers / TLS signature, because it's their worker doing the connection.

pip install curl_cffi

Even easier than spending 15 minutes setting up cloudflare workers.

>The racket is not in the workers themselves, but rather cloudflare both protecting from internet abuse and protecting sites which sell the abuse services. (For example hosting WebStresser) I meant that by giving them more traffic and accepting that as a workaround, we'd be saying "I'm ok with that".

Do you think it's a "racket" for gun shops to sell guns for home defense, but also to sell guns to criminals?


> Even easier than spending 15 minutes setting up cloudflare workers.

You need both in practice. Changing the TLS details won't save you from coming out of the same CGNat as the rest of your city for example.

> for gun shops to sell guns for home defense, but also to sell guns to criminals?

If they know they're selling to criminals who are likely to attack their customers, then of course yes. In practice the overlap is not as trivial so I don't think it really transfers that well. So really "mu, the analogy is not close enough".


> If you have access to the hosting side, you can adjust the bot score for specific connections/clients

Only for Enterprise customers [1].

[1]: https://developers.cloudflare.com/bots/plans/bm-subscription...


Can't you have a normal firefox profile for such cases? Do you have any javascript filters? I bet the issue must be related to configs messing with the JS runtime.


The issue is scummy companies like cloudflare which are causing these issues. If your software is blocking legitimate users then your software is shit at its job. It's not the users fault.


Agreed, but I think the point was that the user has a workaround. Use a standard browser for the like five minutes it might take to unsubscribe from these mailing lists, a one-time operation per business, done.

If on the other hand unsubscribing from mailing lists is not the true use case and we are actually being asked to help a bot bypass safeguards… then Cloudflare is doing a great job here.


>The issue is scummy companies like cloudflare which are causing these issues. If your software is blocking legitimate users then your software is shit at its job. It's not the users fault.

But if you're going out of your way to look suspicious (ie. "I use a heavily customized Firefox config on Linux"), surely you'd agree at some point it goes from "your software is shit at its job" to "it's your fault for looking suspicious"? If you walk into bank wearing a balaclava and get stopped by security, it's not really "security is shit at its job".


Yeah we could start blaming victims.

Maybe we should not be allowed to use software we want to use. Everyone should only be allowed to use windows and a chrome browser variant with no ad blocking. Cloudflare 100% should be allowed to arbitrarily block anyone not using this set up because they are suspicious.


>Everyone should only be allowed to use windows and a chrome browser variant with no ad blocking. Cloudflare 100% should be allowed to arbitrarily block anyone not using this set up because they are suspicious.

Seems like a slippery slope argument, but isn't reflective of reality. They still allow Tor browser to pass, of all things.


It wasn't meant to be taken seriously, I was using it to show the ridiculousness of blaming a user for the shortcomings of cloudflare.

But if you like: the arbitrarily blocked user if not at fault, cloudflare is at fault.


>I was using it to show the ridiculousness of blaming a user for the shortcomings of cloudflare.

That doesn't advance the conversation, or show that cloudflare should be always as fault, as you seem to imply. Even if people are pro privacy/freedom, I think most wouldn't give the individual (as opposed to the security provider) unlimited leeway, as seen in the bank example.


Does "But if you're going out of your way to look suspicious" advance the conversation?


It advances the conversation because it refutes the argument that "It's not the users fault" brought up a few comments ago, by using the balaclavas in bank analogy.


It refutes nothing, it attempts to place the blame for cloudflare incompetence at the feet of a user who has done nothing wrong.


Replied in the other comment. In short: if someone wears a balaclava to a bank, would you say that person also "has done nothing wrong"?


In short, just answering this question (which as stated in the other comment is not the issue), yes that person has t done anything wrong. There is no offence for wearing hats in banks.


>There is no offence for wearing hats in banks.

But banks aren't mandated to admit you either. Just because it's legal, doesn't mean a private establishment has to let you in. When it comes to denying entry, banks are relatively tame. Some establishments go beyond that, by denying entry unless you wear formal clothing, or presenting proof of identity.


This is now completely off topic, well done.


No one said that the user gets unlimited leeway. And we're not talking about that. But you can strawman as much as you like.

We're talking about browsing the internet; and being blocked from doing that because of the incompetence of a "security" company.


>No one said that the user gets unlimited leeway. And we're not talking about that. But you can strawman as much as you like.

Then why are you refusing to address the fact that the OP seemingly went out of his way to seem suspicious, by using a "heavily customized Firefox config on Linux"? There might be legitimate reasons for doing that, and it's not illegal to customize your browser, but those factors are true for wearing balaclavas as well. Why don't you think cloudflare is justified in refusing service to the OP, given that whatever he's doing is probably in the top 0.1% of suspiciousness?


There you go again with your strawman.

You seem to be confusing suspiciousness with using software set up the way you want it set up. There is no fact about the OP going out their way to be suspicious.

The *fact* you and cloudflare seem to think using software set up the way the OP wants is suspicious is the actual issue.

No idea why you seem to be obsessed with balaclavas but the analogy doesn't work, and is simply there to try and make your strawman argument look less like one.

The implicit assumption of you final question is that the OP is in some way acting suspiciously, they are not. They are using their browser set up how they like, and are doing NOTHING wrong. I'll refer you back to my previous point: The *fact* you and cloudflare seem to think using software set up the way the OP wants is suspicious is the actual issue.


>There you go again with your strawman.

Please look up what a strawman is. Disagreeing what counts as "suspicious" is a disagreement, not a strawman.

>You seem to be confusing suspiciousness with using software set up the way you want it set up. There is no fact about the OP going out their way to be suspicious.

By that logic would you say that wearing a balaclava in a bank isn't suspicious either, because it's "wearing clothes you want to wear"?


> The typical straw man argument creates the illusion of having refuted or defeated an opponent's proposition through the covert replacement of it with a different proposition and the subsequent refutation of that false argument, instead of the opponent's proposition

You're not addressing the actual issue but instead replacing it with your twisted take, and then refuting it. In case you missed it you're attempting to make the OPs behaviour out as suspicious. And then having made it suspicious stating the cloudflare is well within their rights because the OP is acting suspiciously. Strawman. And your broken analogy involving a balaclava (now in a bank) feeds into this false narrative.

The OPs use of their software set up how they like is not equivalent to the OP entering a bank in a balaclava.

Cloudflares behaviour is the equivalent of some sort of secret police setting up a roadblock and refusing to let you pass unless you are dressed in a certain way, whilst also stripping you naked and grabbing all you bio data to boot.


>You're not addressing the actual issue but instead replacing it with your twisted take, and then refuting it. In case you missed it you're attempting to make the OPs behaviour out as suspicious. And then having made it suspicious stating the cloudflare is well within their rights because the OP is acting suspiciously. Strawman. And your broken analogy involving a balaclava (now in a bank) feeds into this false narrative.

"not addressing the actual issue but instead replacing it with your [...] take" is literally how analogies are supposed to work. If you disagree with it, you have to point out why it's not applicable. Over the last few comments you have failed to do this.

>The OPs use of their software set up how they like is not equivalent to the OP entering a bank in a balaclava.

Why?

>Cloudflares behaviour is the equivalent of some sort of secret police setting up a roadblock and refusing to let you pass unless you are dressed in a certain way, whilst also stripping you naked and grabbing all you bio data to boot.

I can't tell whether you're not self-aware, or are purposely engaging in bad analogies as some sort of way to get back at me for my balaclava in bank analogy.

Regardless, comparing browser fingerprints to "stripping you naked and grabbing all you bio data to boot" is a stretch. On a theoretical level, your device fingerprint isn't tied to your body, and can be changed, unlike actual fingerprints. You can have multiple devices. Devices can be shared. You can change around settings to yield different fingerprints. At a practical level, most people care far more about being "naked" and "bio data" than people knowing that they're using an iPhone 15 Pro (or whatever), which realistically is what fingerprinting amounts to. Moreover, "setting up a roadblock" is only objectionable because it's interfering with your ability to travel freely. If I want to go to a concert, and a gang sets up a road block on the interstate, my ability to go to the concert is being being impeded by a third party. However, cloudflare isn't a third party. They're contracted as the security vendor by the website you're trying to visit. Therefore, it's less like "secret police setting up a roadblock" and more close to a bouncer not letting you in because he doesn't like the look of you. Since the site is basically private property, private property owners are free to choose who to admit onto their premises, and the site owner has delegated that authority to cloudflare, cloudflare is free to reject you entry for whatever reason, just like a bouncer is free to reject you entry for whatever reason.


You also broke the site guidelines in this thread. Please don't do this, regardless of how provocative other comments are or you feel they are. It just feeds a downward spiral.

https://news.ycombinator.com/newsguidelines.html


I see no point in engaging with you further. You repeatedly attempt to sidetrack the conversation by changing the subject and providing various strawman arguments and bad faith statements. You have nothing valid to add or offer, as proven by you most recent replies. You can continue your disingenuous nonsense if you like but your points are moot since you are simply addressing your own off topic analogies. I'm done feeding the troll. Or corporate shill?


You broke the site guidelines repeatedly and badly in this thread, crossing into swipes and personal attacks. We have to ban accounts that post like this. Fortunately I don't see other cases of this in a quick runthrough of the account's posting history, so this should be easy to fix. If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.


Mobile operating systems with remote attestation (that's both Android and iOS) aren't far off from that with regard to native apps. It doesn't affect the web yet, but Google did propose adding an attestation mechanism to Chrome.


I vaguely remember this from this last year though I can't remember all the details. That's a scary slippery slope.

Of course it'll be presented as a security feature, because users are dumb, whilst also allowing vendors to lock you into their ecosystem; similar to how passkeys are currently being push by these same companies.


Cloudflare challenges seem to be becoming more and more frequent on my general internet use. Yep, "Cloudflare loop" is a thing. No, I'm not going to download and install a different web browser, dump all my cookies, or whatever other nonsensical "solution" they recommend.

I've become to hate Cloudflare with a seething passion.


I ran into this, or something similar recently when our main connection went down (solar powered) and we switched to Starlink. Due to Starlink NAT issues I had tunneled our traffic to to a box colocated in a data center. This broke a number of web sites in weird ways. Became so annoying that I ended up bringing up a tunnel to our office in town to get back to the regular IP we used. Weird problems went away.


Slightly off topic, but Microsoft ones are even worse - when I tried to sign up to OpenAI/get a new Microsoft account, the captcha were so difficult that it took me 5 minutes to solve (unsuccessfully). As a libre wolf user with very strict settings, I think privacy-aware users bear the externalities of this bot vs server arms race.


Vaguely related, youtube is lately doing a lot of unnecessary forced logouts & reconfirm password. I'm literally on a static IP. On the same computer & browser. With the same cookies. Not accessing anything particularly sensitive. There is no way in hell they don't know precisely who I am & that its me.


Is it possible your account is being attacked in the background?


Possible I guess. It is yubikeyed though so would need to be a pretty sophisticated attack


I spent a few days agonizing over this same problem, and the culprit turned out to by my user-agent modifier extension.


What I don't understand is why you have to protect areas that require login so harshly?

If I can log in, especially with 2-factor, you can safely assume I am not a bot, or you have a larger problem.

If I have entered bad credentials 5+ times, okay, you can start backing me off or challenging me.

What am I missing? Fail2ban has been around a long time.


Problem is that a significant chunk of the technology industry still relies on "engagement" as its business model. The objective of slapping an overzealous bot protection system isn't to protect high-risk endpoints like logins/etc, it's to ensure a human is "engaging" and human time is being wasted by making even legitimate automated usage impossible.

From their perspective, the blocking of power users with unusual setups is actually a happy coincidence, as those are unlikely to "engage" with the product in the desired way (they run ad & spyware blockers, don't fall for dark patterns, and are more likely to fight back if they get defrauded by the corporation).


40% of the internet’s traffic now is bots, with about half of those being malicious. Fail2ban is decent for a very small DDoS, but useless for one with any substance, and also useless against bots scraping data or probing for weaknesses.

Also remember, especially on AWS, bandwidth is expensive. A CDN cache + blocking bots = big savings.


> What am I missing? Fail2ban has been around a long time.

Modern threat actors can spread requests out over large pools of source IPs. Rate limiting login attempts by IP isn't an effective means of preventing credential stuffing attacks.


I use Whonix quite a lot, Most of the internet is unusable since i get into the "check the box" loop.


The challenges are configurable by cloudflare’s customers. The challenge can either be from turnstile, which is a captcha replacement service that websites use on their own pages, or a cloudflare CDN security setting that will block access to pages until a challenge is passed. It is not clear which one the original poster means.

Cloudflare’s customers can largely disable these and rely on other means of detecting bots.

In the case of turnstile, it has three modes, two of which are entirely automatic and work by interrogating the web browser, with the other requiring a client:

https://developers.cloudflare.com/turnstile/concepts/widget/

Cloudflare CDN’s security setting on its free tier also has an essentially off setting that will basically eliminate challenges when browsers accessing pages protected by cloudflare unless there are exceptional circumstances. I believe it can be fully turned off for the enterprise tier.

Whenever I configure cloudflare for a website, I always turn off challenges since they are annoying to users. There is an interesting write up about how cloudflare’s bot detection works here:

https://blog.capmonster.cloud/en/blog/web-scraping1/how-clou...

Note that I have yet to use turnstile, so I am speaking from documentation I read rather than from actual experience with it. I have used cloudflare’s CDN and I am speaking from experience with it.

Anyway, the website author is the one that should be blamed here.


Try creating a cloudflare.com account and stay logged into it. I.e. every few days go into the cloudflare dashboard.

Don't know if it will help but they use lots of methods to see if you are hostile, and being logged in and authenticated with them can't harm


Just tried to disable User-Agent in Firefox by setting general.useragent.override to an empty string and Cloudflare captcha become impenetrable. Cloudflare actively blocks attempts to improve privacy :(


You don't want that to be an empty string - you would be one of the few people in the world with that value and thus easy to identify. You instead want that string to be exactly the same as everyone else in the world (no matter what the real User-Agent really is). there should be about 50 different contents of the entire header possible for everyone in the world.


Mess? I got a 8 try guess Ilol to try in 5 tries... in an indistinguishable font. Ooo... Im gonna fail that one...

I am good at this stuff, and "Cloudflare challenges have made large portions of the web unusable for me" too.


Same here, but Cloudflare's captchas in particular are actually the easiest to pass in my experience. Google's ones are the killers. But yeah everything has a captcha if you're using a VPN or Firefox.


you're not welcome. is their message.

not a single mention of advertising on all these comments.

those captcha are not against bots. bots are only one item in the broader category they block. you, an unmonetizable user, is another.

cloudflare et all have the "marketplace conundrum". they need to provide value to both sides, and for the site they do this by blocking hard to monetize traffic. that means traffic that won't generate high yield on ad networks those sites care about.


Try some popular user agent strings first before concluding that something else like TLS fingerprinting is the problem. Sometimes an acceptable UA string is all it takes.


My local TV station's website refuses to allow my to view their page and instead presents an a modal that cannot be blocked accusing me of using an ad blocker. The funny thing is that only happens on a mobile device using the default browser with no extensions. When I visit the same site on my laptop with uBO, the site is viewable with no blocking modals.

Sometimes you miss what you were aiming for I guess


Just in time: https://doom-captcha.vercel.app


This is pretty tough on mobile


I would recommend using FlareSolverr as a proxy in your browser to bypass the clownflare's captcha


It seems that if you use Firefox with an adblocker then cloudflare spam is all you see. Though I have experienced this in plain Firefox too.

Cloudflare are a scummy company trying to force you to use one browser and view all ads.


It can't be just that. I use Firefox on Linux with ublock origin, strict tracking protection, and clear cookies on exit, and I've never ever seen a cloudflare challenge. Not even on sites with that "verifying your browser" page enabled.


Maybe you're right, I see it all the time. Assume cloudflare do other dumb stuff too then like up ranges and just being generally crap at their jobs.


Set up your own vpn on AWS ec2. It will bypass the vpn blocks they have. Problem solved.


From the other perspective, I use Cloudflare for DNS and HTTPS certificates. Having an alternative that would cover these two use cases without the need for manually running letsencrypt would be enough for me to switch.

I don't want to think about HTTPS, my websites are low risk, mostly static pages (and there are tens of them).


well, looks like a business opportunity - a service using AI to automatically pass the challenges like this so the people like the original poster could, for a small service subscription fee, use the Internet hassle-free again.


I've honestly only experienced the opposite; their captcha is reasonably easy to bypass, and I've successfully automated access to a few sites "protected" by the Cloudflare captcha (behind a VPN, no less).

> I use a heavily customized Firefox config on Linux.

If you really care about privacy, you should blend in to look like everyone else. Avoiding being tracked raises alarm bells. You have to let them track something; but no one ever said it had to be you.


> I use a heavily customized Firefox config on Linux.

I also use a (not-so-heavily) customized Firefox config on Linux. I also see repeated abuse of my network activity by Cloudflare.


CF issue, or site programming issue.?


You’re using a dirty IP and not using Apple Safari, which has solved this via Private Access Tokens. Move out of the sticks.


Now every couple of minutes when scrolling through Reddit, red "Network issue" tab appears. Some comments don't load at all, some are labeled "deleted" even though they aren't. Refreshing the page usually does the trick, but I hate this new experience.

I guess they're just protecting themselves from bots, and I look like a bot in their eyes.


I found a GitHub captcha to be unsolvable. That captcha properly stressed me out.


yes yes yes yes yes yes yes. I nearly wrote a borderline hit piece of cloudflare challenges because of this bullshit, but instead I gave into their games and repealed my privacy (only for niche cases mostly), likewise there's no solution for me either and it's just, like you said, some other variant of "too bad, so sad".

reply


Stop acting shady.


Cloudflare's —and most similar services'— stance here comes from these VPN funnelling not just people like you, but also attackers. It's untrustworthy traffic from their perspective.

Use a VPN but use a normal network. VPN back to your home, your office. Your traffic will probably take a throughput and latency hit but it looks like real residential traffic, and that's a lot less sus.


but then all of your traffic comes from a single IP which is eventually associated with your identity. this defeats one of the core purposes of using a VPN to circumvent surveillance capitalism.


I'm not saying you're wrong, but in the context of travel, I would suggest most people use the VPN because they don't trust the networks they're connecting to, more than wanting to avoid surveillance, which would apply without the travel component.

I also can't think of one of the popular VPNs that get heavily advertised that I'd trust to actually protect my privacy.


What do you mean impassable challenge...? Why isn't it passable? Are you a robot?


The challenge is a small javascript program that checks the execution environment is consistent with a real browser. For instance, if your user agent says it's chrome, but it's missing features that'd normally be supported by chrome, it'll fail you. The OP mentioned "heavily customized Firefox config", so he might be doing stuff like this that makes his browser look suspicious.


From website perspective, yes. GP is likely using extreme ad-blocking and/or coming from regions where tons of bots and/or unwanted traffic are also from. In those cases, some/many human users could be misidentified as bots with little incentives to website admins to rectify.

And it's discriminatory, yes.


[flagged]


Sadly, we probably all are LLMs/bots on the internet at this point, just talking to one another. The real humans have all become fed up and are now mostly off fishing by a lake.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: